acceptodds
Under review as a conference paper at ICLR 2027

AdaptArena: Evaluating Test-Time Personalization of Web Agents

Abstract

Large language model (LLM) agents have demonstrated strong performance on complex web navigation tasks, yet they remain brittle in real-world settings where user intentions are underspecified and preferences are heterogeneous. In practice, users rarely provide explicit profiles, requiring agents to infer latent preferences from implicit signals. Despite its importance for deployment, this problem setting is largely underexplored in existing benchmarks. To address this gap, we introduce AdaptArena, a benchmark for evaluating test-time personalization of web agents via implicit preference inference. AdaptArena consists of 480 tasks, featuring both single-preference and complex compositional (double-preference) scenarios. Each evaluation task must be solved by retrieving and leveraging the most relevant historical user trajectory that implicitly encodes the target preference. In addition, we introduce AdaptiveAgent, a retrieval-based framework for standardized evaluation of implicit preference inference. Experiments reveal a substantial performance gap: while oracle agents with access to ground-truth preferences achieve an 82.92% success rate, state-of-the-art LLM agents using our framework reach only 15.62%. Furthermore, we find that correctly inferring user preferences is necessary but not sufficient for task success, as execution failures in downstream web interactions remain a significant bottleneck even when agents align with the target preference. These findings highlight implicit preference inference and robust action grounding as key challenges for deploying reliable, user-facing web agents.

Primary area:
datasets and benchmarks
Keywords:
Web Agents, Personalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kSaqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the challenge of providing personalized services when user intent is unclear and preferences vary. It introduces the AdaptArena benchmark and the retrieval based AdaptiveAgent framework to evaluate how agents infer and use implicit preferences from past interactions. The results show that current agents still have clear weaknesses in preference inference and web task execution. Even when agents correctly understand user preferences, they may still fail to complete tasks. The study identifies implicit preference inference and reliable execution as key challenges for personalized web agents.

Critical strengths
  1. The paper introduces the AdaptArena benchmark. It covers multiple web environments and settings with one or two preferences. Agents need to infer user preferences from past interaction trajectories when instructions lack detail. This evaluates their ability to personalize their behavior at test time.
  2. The paper proposes the AdaptiveAgent framework. It retrieves past trajectories, infers hidden preferences, and uses them to guide task execution. This provides a simple and reproducible standard process. The results show that its performance remains much worse than when agents directly receive the true preferences.
  3. The paper highlights key challenges in personalization. All models perform worse in settings with two preferences, showing the difficulty of handling combined preferences. Correct preference inference also does not guarantee task success, as action execution remains a major bottleneck.
Critical weaknesses
  1. All models rely on GPT-5-nano to extract preferences. If this module makes mistakes, even strong models may struggle to complete the task. The paper needs to compare more models on preference extraction.
  2. The paper focuses on 10 personas, and models generate historical trajectories based on clearly defined preferences. It does not verify whether these 10 personas cover the complex and changing behaviors and traits of real users.
  3. The history is small. Each user has only 12 historical trajectories. This setting does not test whether the system handles large amounts of irrelevant records, repeated records, and similar but unrelated behavior as the history grows.
  4. Does the evaluation consider the performance of models together with their agent harnesses, such as Codex and Claude Code? Including these systems can further strengthen the paper.
Questions for authors

See above.

Reviewer_9jisOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether web agents can infer a user's preferences from past interaction trajectories, without an explicit profile. The authors introduce AdaptArena, built on three WebArena environments, with 120 history trajectories (10 personas x 12 preference categories) generated by an LLM agent given explicit preferences, and 480 underspecified tasks with one or two target preferences. They also propose AdaptiveAgent, which retrieves screenshot prefixes from two relevant history trajectories with CLIP and has GPT-5-nano infer the preference from them as a short summary for the web agent. Across six backbones, AdaptiveAgent reaches at most 15.62% success, compared with 82.92% for an oracle given the gold preference.

Critical strengths
  • Relevant problem. Inferring preferences from past behavior is closer to real deployment than relying on explicit profiles or personas, and splitting failures into retrieval, alignment and execution (Sections 4.2 and 4.3) is a useful diagnostic.
Critical weaknesses
  • The main finding does not measure the backbones' ability to infer preferences. In AdaptiveAgent, preference inference is done by the same CLIP retriever and GPT-5-nano extractor for all six backbones, which only receive a short natural-language summary. The correct trajectory is among the two retrieved ones 71.88% of the time, and the same backbones reach 40.83% to 82.92% when given the gold preference (Table 1), so the 15.62% result mainly reflects the shared retrieval and extraction stage. The paper also presents AdaptiveAgent both as a standardized evaluation harness and as its own method, yet it is below the Reconstructed Profile baseline (up to 17.71%) for every model. No setting lets the backbone read the user history itself, so the benchmark's main conclusion is not supported.

  • The synthetic histories cannot test preference inference as intended. Each preference is encoded by exactly one short LLM-generated trajectory (9.78 steps on average), and all ten personas share the same 12 categories, so the task is closer to matching a task to the right category than to inferring preferences from noisy behavior. Appendix A lists this as a limitation, since the histories are built from explicitly specified persona traits rather than real user behavior, and fixing this requires new data rather than new analysis. All results are also reported without variance even though models are sampled at temperature 1.0, while some conclusions rest on gaps of about 10 out of 480 tasks (Table 4).

Questions for authors
  • How do the backbones perform when they infer preferences themselves, e.g., with the full user history in context or with the backbone replacing GPT-5-nano as the extractor?
  • What is the run-to-run variance of the main results in Table 1 and the ablation in Table 4?
  • How were the deployment tasks validated as underspecified without history and unambiguous with it, and what was the inter-annotator agreement?
Reviewer_D8QKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AdaptArena, a benchmark for evaluating the ability of web agents to infer implicit user preferences from historical interaction trajectories and apply them to new tasks at test time. The benchmark comprises 480 single- and double-preference tasks across three web environments, with 120 agent-generated historical trajectories covering ten synthetic personas and twelve preference categories.

The authors also propose Adaptive Agent, a retrieval-based framework that identifies relevant historical trajectories, extracts latent preferences, and incorporates them into downstream web-agent execution. Experiments across six backbone models reveal a substantial performance gap between Adaptive Agent (up to 15.62% task success) and an oracle setting with access to ground-truth preferences (up to 82.92%). Further analyses examine compositional preferences, retrieval quality, and the gap between preference alignment and successful task execution.

Critical strengths

(1) AdaptArena addresses an important problem: executing underspecified web tasks using user preferences inferred from interaction history. The controlled single- and double-preference tasks provide a useful setting beyond generic web-task success evaluation.

(2) The comparisons across six backbones and different information conditions, together with category-level retrieval analysis and cross-user history assignment, expose informative differences between user-context availability and end-to-end task performance.

Critical weaknesses

(1) Limited differentiation from existing personalized web-agent benchmarks. The central problem studied by AdaptArena—inferring implicit user preferences from historical interactions to resolve underspecified web tasks—has already been explored by closely related benchmarks such as Persona2Web. Persona2Web also considers implicit user histories, ambiguous queries, different history-access strategies, and fine-grained personalization evaluation. PersonalWAB similarly incorporates historical web behaviors and task-specific memory retrieval.

AdaptArena introduces some useful differences, particularly screenshot-based interaction trajectories, controlled browser environments, and double-preference tasks. However, it is not sufficiently clear what additional evaluation capabilities these designs provide compared with existing benchmarks. For instance, the lower success rates on double-preference tasks may partly result from increased retrieval requirements, longer interaction sequences, or greater task difficulty, rather than compositional preference reasoning itself.

I would like to see a more concrete comparison with prior benchmarks, explaining what AdaptArena measures that existing evaluation settings do not adequately capture.

(2) Unclear definitions and inconsistencies in the diagnostic evaluation. Some reported statistics are difficult to reconcile. There is ambiguity in the retrieval analysis. The 63.3% Wishlist-vs.-Cart alignment in Table 15 is described in the text as conditional on successful retrieval, while the table note appears to report a category-level average. The Oracle setting and Recall@2 definition for double-preference tasks would also benefit from clarification.

More generally, Preference Alignment is judged from executed interaction trajectories rather than directly from the preference extractor's output. As a result, it is difficult to distinguish incorrect preference inference from downstream execution failures using the current metrics alone.

Questions for authors
  1. Could the authors report the raw counts of aligned, aligned-and-successful, and aligned-but-unsuccessful episodes, together with the number classified in Table 3 for each backbone? In particular, how are the apparent discrepancies for GPT-5-mini and Gemini-3-Flash explained?
  2. Can the authors directly evaluate GPT-5-nano's extracted preference summaries against gold preferences and report joint statistics for extraction correctness, behavioral alignment, and task success?
  3. For double-preference tasks, does the Oracle receive both gold preferences? Is Recall@2 defined as any-hit or all-hit? Is the 63.3% Wishlist-vs.-Cart alignment computed over all episodes or only retrieval-successful episodes?
  4. Could the authors clarify the distinctive evaluation capabilities of AdaptArena relative to closely related benchmarks such as Persona2Web, particularly regarding interaction-trajectory evidence, compositional tasks, and diagnostic metrics?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.