- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces UserToolBench, a benchmark for evaluating personalized decision-making in tool-use LLMs. It tests whether models can infer user preferences from interaction history and make appropriate tool calls when the current request lacks important information. The benchmark includes 10 user profiles, 1,065 dialogue turns, and 170 tools. Experiments on nine LLMs show that current models still struggle with personalized tool use, particularly in multi-tool tasks and situations with missing information.
Critical strengths
-
The paper studies a meaningful problem for personalized AI assistants. The profile-hidden evaluation setting is interesting, as it requires models to infer user preferences from interaction history rather than directly accessing a user profile.
-
The authors evaluate a range of LLMs and consider different task types. Reporting both exact and relaxed accuracy also helps distinguish strict trajectory matching from task completion.
Critical weaknesses
-
Both the user simulator and the reference trajectory generator have access to the full user profile. I am concerned about two possible problems. The simulator might reveal the user's preferences too explicitly, making the task too easy. On the other hand, the generated history might not provide enough information to infer the preferences used in the reference trajectory, or several interpretations might be equally reasonable. It is unclear how the authors avoid these cases.
-
Although LLM-based user simulation is common, I am not fully convinced that these generated conversations reflect how real users interact with assistants. In particular, conditioning the simulator on a fixed profile may produce overly consistent user behavior and unnatural expressions of preferences.
-
Real users do not always have stable preferences. Their preferences may change over time or depend on the situation. The authors consider this in Appendix D.3, but only through a preliminary experiment. I think the main benchmark still makes a rather strong assumption about preference stability.
-
Table 1 describes UserToolBench as a long-horizon benchmark. However, the dataset contains 1,065 dialogue turns across 300 topics, averaging only 3.55 turns per topic. I understand that this does not necessarily reflect the full history length, but the paper provides little information about how long these histories actually are or how far back the relevant preferences appear. This makes the long-horizon claim less convincing.
-
Appendix E.1 describes a human verification interface, but the paper lacks details about the annotation process. For example, how many annotators were involved? Were samples independently reviewed by multiple annotators? What was the inter-annotator agreement? Given the reliance on synthetic data, I would like to see more evidence of the dataset's quality.
-
I did not find an experiment where the evaluated LLM is given the complete user profile. Without this baseline, it is difficult to tell whether models fail because they cannot infer user preferences from the interaction history, or because they cannot make appropriate tool calls even when those preferences are known.
Questions for authors
-
How do the authors prevent the user simulator from revealing preferences too explicitly? Have they checked whether the reference decisions can actually be inferred from the interaction history, especially when multiple interpretations are possible?
-
Have the authors compared the simulated conversations with real user interactions to evaluate how natural and realistic they are?
-
Could the authors discuss how the benchmark handles changes in user preferences, beyond the preliminary experiment in Appendix D.3?
-
Could the authors report the distribution of history lengths, in both turns and tokens? It would also be useful to know how far back the relevant preference information appears.
-
Could the authors provide more details about the human annotation process, including the number of annotators, inter-annotator agreement, and sample acceptance rates?
-
Have the authors considered an oracle-profile baseline where the model directly receives the user profile? I think this comparison would help clarify whether the main difficulty comes from preference inference or tool-use planning.