- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a benchmark for personalized tool use in long-horizon, multi-turn travel planning. The paper builds synthetic personalized tool-use trajectories from realistic travel scenarios, with rich user profiles, subgoals, multiple tools, and human-validated dev/test sets. It then fine-tunes open models with LoRA and compares them with zero-shot baselines. Fine-tuning substantially improves personalized tool calling, but the hardest part is identifying which preferences in a full user profile are actually relevant. Models also perform much better on retrieval tasks than on execution tasks such as booking.
Critical strengths
This paper design a personalized multi-turn long-horizon multi-tool use benchmark with rich user profile and propose a training set. Evaluation show the benefits of training data and analysis demonstrates the failure mode of LLMs.
Critical weaknesses
- The long-horizon part is next-tool-call prediction in history and doesn't test the planning and agentic capability of LLM, while nowadays agent test is more common.
- The claimed gains are driven largely by retrieval-oriented calls, while fine-tuning substantially degrades actionable tool use such as booking and calendar operations. There is no solutions for this.
Questions for authors
Is there any method to improve both retrieval and actionable tool use?
Response to Reviewer nHFn
We thank the reviewer for providing feedback on our paper. We address the concerns below.
Long horizon evaluation
Please see our general response on long-horizon evaluation, where an adaptive mode lets models choose their own next actions across whole scenarios, bringing our protocol closer to agentic evaluation and revealing clear failures in current models. Although models often follow their own valid order (23–39% of accepted calls differ from the reference), none completes a scenario, and models typically fail within their first two calls.
Fine-tuning gains and ways of improving
We agree that fine-tuning improves retrieval while often degrading actionable tool use. Our error analysis suggests that the imbalance in the training data contributes to this behaviour: retrieval calls dominate the generated trajectories (Appendix A), and fine-tuned models frequently select search operations instead of booking actions.
Potential remedies include rebalancing or upsampling execution examples and weighting the training loss by call type. Our automatic data-generation pipeline supports such interventions, although we have not yet evaluated them. The purpose of ToolTailor is precisely to expose these trade-offs, which aggregate tool-use metrics would conceal. We will clarify that improving both retrieval and execution remains an open challenge rather than an established result of our training approach.