acceptodds
Under review as a conference paper at ICLR 2027

ToolTailor: Personalised Tool Use in Multi-Turn Travel Planning

Abstract

Tool-augmented language models have become the dominant paradigm for task-oriented assistants, yet existing benchmarks rarely evaluate whether these models integrate individual user preferences into their tool calls. We introduce ToolTailor, a benchmark for personalised tool use in travel planning, together with a synthetic data generation pipeline that produces personalised tool-use trajectories at scale. ToolTailor is the first benchmark to combine long-horizon, multi-turn tool use with rich user profiles, requiring agents to satisfy individual preferences while orchestrating multiple tools across a task. Tasks are seeded from travel-planning scenarios collected from web forums, and the development and test splits are human-validated. Evaluation with state-of-the-art LLMs shows that personalised tool use remains challenging: models fail to identify and incorporate relevant user preferences into tool calling. Supervised fine-tuning on our synthetic trajectories improves preference integration by up to 35 F1 points when the relevant preferences are provided directly, while also improving schema compliance. When the model must instead identify them in the full user profile, all four fine-tuned open models still outperform three proprietary models (Gemini-3.6-Flash, GPT-4o-mini and GPT-5.6-Terra) prompted zero-shot.

Primary area:
datasets and benchmarks
Keywords:
too use, personalization, goal-oriented dialogue, benchmarking, NLP datasets, evaluation methodologies, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nHFnOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for personalized tool use in long-horizon, multi-turn travel planning. The paper builds synthetic personalized tool-use trajectories from realistic travel scenarios, with rich user profiles, subgoals, multiple tools, and human-validated dev/test sets. It then fine-tunes open models with LoRA and compares them with zero-shot baselines. Fine-tuning substantially improves personalized tool calling, but the hardest part is identifying which preferences in a full user profile are actually relevant. Models also perform much better on retrieval tasks than on execution tasks such as booking.

Critical strengths

This paper design a personalized multi-turn long-horizon multi-tool use benchmark with rich user profile and propose a training set. Evaluation show the benefits of training data and analysis demonstrates the failure mode of LLMs.

Critical weaknesses
  1. The long-horizon part is next-tool-call prediction in history and doesn't test the planning and agentic capability of LLM, while nowadays agent test is more common.
  2. The claimed gains are driven largely by retrieval-oriented calls, while fine-tuning substantially degrades actionable tool use such as booking and calendar operations. There is no solutions for this.
Questions for authors

Is there any method to improve both retrieval and actionable tool use?

1 comment
AuthorsOpenReview ↗

Response to Reviewer nHFn

We thank the reviewer for providing feedback on our paper. We address the concerns below.

Long horizon evaluation

Please see our general response on long-horizon evaluation, where an adaptive mode lets models choose their own next actions across whole scenarios, bringing our protocol closer to agentic evaluation and revealing clear failures in current models. Although models often follow their own valid order (23–39% of accepted calls differ from the reference), none completes a scenario, and models typically fail within their first two calls.

Fine-tuning gains and ways of improving

We agree that fine-tuning improves retrieval while often degrading actionable tool use. Our error analysis suggests that the imbalance in the training data contributes to this behaviour: retrieval calls dominate the generated trajectories (Appendix A), and fine-tuned models frequently select search operations instead of booking actions.

Potential remedies include rebalancing or upsampling execution examples and weighting the training loss by call type. Our automatic data-generation pipeline supports such interventions, although we have not yet evaluated them. The purpose of ToolTailor is precisely to expose these trade-offs, which aggregate tool-use metrics would conceal. We will clarify that improving both retrieval and execution remains an open challenge rather than an established result of our training approach.

Reviewer_FXbhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

ToolTailor is a benchmark and synthetic-data pipeline for personalised tool use in multi-turn travel planning. Each scenario pairs an LLM-generated user profile (demographics, lifestyle preferences, 2–3 preferences per tool in a personal toolset) with 27 simulated tools across six domains, a plan drawn from eight fixed subgoals, and a multi-turn reference trajectory; the opening user query omits the stored preferences so the model must retrieve them. Ten Reddit queries serve as few-shot seeds for 15 themes; Qwen3-8B generates the training split and Qwen3-14B the dev/test splits via an action-LLM / user-LLM / tool-simulator / completion-checker loop. Dev and test are crowd-validated (single annotation, expert resolution); test trajectories are expert-edited with Claude Code assistance. Evaluation is step-wise next-tool-call prediction scored as ⟨tool, function, argument, value⟩ tuples under strict or dependency-aware matching, with separate personalised-argument and schema-compliance metrics, in three settings (all preferences and tools; relevant preferences; relevant preferences and tools). Splits are 82 / 10 / 331 scenarios (2,313 / 67 / 1,945 instances). The paper reports that LoRA fine-tuning Gemma3-4B and Qwen3-4B/8B/14B on the synthetic split improves F1 by +3 to +19 in the hardest setting and up to +35 when relevant preferences are given, that all four fine-tuned models beat zero-shot Gemini-3.6-Flash, GPT-4o-mini and GPT-5.6-Terra, and that identifying relevant preferences is the main bottleneck.

Critical strengths
  • Attempting a general formulation of personalised tool use, with training rather than only evaluation. Treating preferences as something that must shape tool selection and argument filling across a long task, and then showing that synthetic trajectories can be used to train for it, is a worthwhile direction that most benchmark papers in this space stop short of. The subgoal analysis also surfaces a useful negative result the authors report honestly: fine-tuning on the synthetic data breaks booking (Table 21: Qwen3-4B 35.1 → 6.7 F1).

  • The evaluation machinery is careful. Dependency-aware matching alongside strict matching, personalised-argument metrics separated from general tool metrics, schema-compliance diagnostics, three seeds with reported variance, and subgoal / function-type breakdowns (Tables 17–21). Human validation is reported with unusual transparency (Table 11 correction rates, recruitment funnel, pay). These are reusable even if the specific artifact needs rework.

  • Paper was easy to read and follow, but the difficulty in understanding the nature of the work, and some claims are discussed in weaknesses. It is helpful that the authors shared code and data.

Critical weaknesses
  • An entire body of directly relevant work is absent, and the novelty claims depend on that absence. The abstract calls ToolTailor "the first benchmark to combine long-horizon, multi-turn tool use with rich user profiles" and Table 1 compares against seven benchmarks, none from travel planning. Uncited and directly overlapping: TravelPlanner (Xie et al., ICML 2024) — the standard travel-planning tool-use benchmark, with constraint types this paper's Table 6 largely duplicates; TravelPlanner+ (Singh et al., EMNLP 2024 Industry) — a personalised version of TravelPlanner with user models, i.e. this paper's exact premise two years earlier; TripCraft (ACL 2025), TripTailor (ACL 2025 Findings), RETAIL (EMNLP 2025) and UserBench (NeurIPS 2025) — personalised or implicit-preference travel planning, the last multi-turn and tool-augmented; PTBench (Huang et al., arXiv 2505.04072) — defines personalised tool invocation as tool preference among functionally similar tools plus profile-dependent argument filling, provides a synthesis pipeline, and fine-tunes open models on it, which is this paper's contribution list; ToolSpectrum (ACL 2025 Findings) and PEToolBench (ACL 2025 Findings) — further personalised tool-use benchmarks; τ-bench (ICLR 2025) and AppWorld (ACL 2024) — multi-turn, long-horizon, multi-tool with simulated users. ETAPP, which is cited, already models users as a profile plus tool-utilising preferences, the structure presented here as distinctive, yet Table 1 marks it partial. The paper never explains why a new simulated environment was built rather than adding profiles to TravelPlanner's existing sandbox, which is what TravelPlanner+ did. Whether this reflects unfamiliarity with the field or a decision not to engage with it, the result is the same: a reader/reviewer cannot locate the contribution. What remains after positioning — the subgoal structure, the step-wise protocol, the travel instantiation — is a modest increment, and the scale (82 training scenarios, 10 dev scenarios) does not compensate. This is the primary driver of the rating.

  • The central empirical claims are undermined by the paper's own tables.

    • Same-family reference confound. Zero-shot Qwen3-4B (25.61 F1) and Qwen3-14B (23.50) already beat Gemini-3.6-Flash (22.02), GPT-5.6-Terra (20.10) and GPT-4o-mini (11.57) in the hardest setting (Table 17), before any fine-tuning. The test references were generated by Qwen3-14B. The "fine-tuned open models beat proprietary models" claim (L399–402) is therefore evidence of reference-style matching, not personalisation learning. Performance is also non-monotone in Qwen size in both regimes (zero-shot 4B > 14B > 8B; LoRA 14B > 4B > 8B).
    • The "relevant preferences" setting leaks the answer. Appendix E defines relevant preferences by the reference target tool, and a separate LoRA is trained per setting (F.1, L1623–1625), so the +35 F1 and 85–90% tool match are partly learned leakage. Restricting the toolset further lowers tool match for every fine-tuned model (Table 17: Qwen3-8B 89.7 → 76.9, Gemma 84.0 → 76.9), which is only consistent with that explanation. (It is possible my understanding about this one point is incorrect, will to discuss during rebuttal)
    • Proprietary baselines are trained-vs-zeroshot. They miss required arguments in 14–34% of calls (Table 19); the paper never says whether native function calling was used, and no few-shot, CoT or self-correction style baselines are run to provide a concrete comparison to the LoRA experiments.
    • Section 5 misreports its own tables. "13–30 F1" gains in Relevant Preferences & Tools (L392–393) are actually 9.4–25.8. "Calendar gains remain limited" (L465), but Calendar falls 11.9 for Qwen3-8B and 4.3 for Qwen3-14B (Table 21). "Consistently improves retrieval" (L453), but Activities falls 11.7 for Qwen3-8B. "Models handle retrieval well but struggle with execution" (L449–452, L480–482), but every zero-shot model — open and proprietary — scores higher on actionable than retrieval (Table 20); it is fine-tuning that reverses this.
    • No plan-level or error analysis. The evaluation never scores the final plan, checks a constraint, or reports anything on Test Full despite L319–322. There is no breakdown of what a failed call looks like, no profile / preference / constraint / theme distributions, and no qualitative examples of failures.
Questions for authors

Clarification:

  • How does ToolTailor differ from TravelPlanner+, PTBench, UserBench and ETAPP in task formulation, and how is this formulation diferent from the tens of seminal travel planning as agentic benchmark works since 2024. A revised Table 1 covering the works above would help.
  • Does the "relevant preferences" subset reveal the reference tool? What are tool match and F1 when relevant preferences are selected by tool category or active subgoal instead?
  • Was a separate LoRA trained per evaluation setting (F.1, L1623–1625)? If so, were the Relevant-Preferences LoRAs trained on the leaky input?
  • How were the proprietary models invoked (native tool-calling API vs. text schema)? What explains 20–34% missing required arguments for GPT-5.6-Terra and GPT-4o-mini (Table 19)?
  • How is trajectory validity measured — i.e. that a reference trajectory actually satisfies the user's request, profile and constraints? Validation covers the query, scenario setting and subgoal list (Section 3.2.4), and test trajectories are expert-edited, but nothing checks the final plan against the profile or the Table 6 constraints. Is there any plan-level correctness signal?
  • What does "execution" failure mean when tools are not executed (L309)? Table 20's "Actionable" rows score booking and calendar calls by tuple match like everything else. If so, the imbalance argument at L466–470 needs spelling out: an under-represented class explains a model not learning to predict booking calls, but the drop is on calls it does predict. What does the fine-tuned model emit at booking steps — a retrieval call, a wrong function, a hallucinated identifier from an earlier tool output? A confusion breakdown over the four tuple slots would settle this.
  • Why does fine-tuning leave Qwen3-14B's schema compliance unchanged (L404–405)? Table 19 shows the mechanism — name precision rises 87.9→99.4 but type precision falls 86.1→82.7 — which implies the training split contains systematic type errors that the model learns. Which argument types and which tools, and is there a schema-error breakdown for the other models, including the proprietary ones?
  • Given that the crowd changed 62–86% of items and the expert changed 24–63% of those (Table 11), with a predictable pattern (missed execution-phase subgoals), why was this not fed back into the automatic generation and repair for the training split (L265–267)? The paper notes train is noisier and stops there.
  • How was annotation workload distributed? D.2 reports 66–73 passing annotators per project and up to five 10-item assignments each, but not how many items each actually annotated, so it is unclear whether the 341 validated scenarios were spread across many annotators or concentrated in a few. Please also reconcile the three "per instance" denominators in Table 11 (414/0.84 ≈ 493, 193/0.62 ≈ 311, vs. 341 scenarios).
  • What are the bucket sizes in Figure 3 / Tables 22–23?

Suggestions — would strengthen a revision:

  • Report results on Test Full and at least one trajectory- or plan-level measure (constraint satisfaction, LLM-judged plan quality against the profile), since step-wise matching cannot show that correct calls produce a usable plan.
  • Add an analysis section: profile, preference, constraint-type and theme distributions across splits; qualitative failure examples; the train/test shape gap (5.4 vs 2.7 calls per subgoal, Table 2); the near-absence of flights (99 of 5,771 test steps, Table 4).
  • Describe user-profile generation in the main paper and address bias and stereotyping in generated personas (demographic distribution, how occupation/age/dietary attributes are sampled, any filtering). This is currently mentioned only as a high-level limitation; the Ethics section covers privacy only.
  • Use the expert-edit pattern in Table 11 to improve automatic repair for the training split, and rebalance retrieval vs. actionable calls.
  • Enlarge the dev split; 67 instances with 4.1 subgoals/scenario (vs. 6.4 in test) is a thin basis for selecting 12 LoRA checkpoints.
  • Run at least one prompting baseline for the proprietary models (native function calling, few-shot via truncated history, or CoT before the call) so the comparison is not trained-vs-untried.
  • State in the AI-use section that the dataset, user simulator, tool simulator and completion checker are all LLM-generated (Qwen3-8B/14B).
3 comments
AuthorsOpenReview ↗

Response to Reviewer FXbh (Part 1)

We thank the reviewer for their detailed feedback. We address the main concerns below and will incorporate the suggested corrections and clarifications in the revision.

Related work

Given the space limitations, our related work focused on tool-calling benchmarks. We will add a section on travel-planning agents, cover the missing benchmarks and extend Table 1 (Thank you for the pointers!).

Travel-planning benchmarks score final itineraries, not tool calling capabilities of LLMs. TravelPlanner+, TripCraft and TripTailor give all relevant information in the prompt, and in TravelPlanner and UserBench tool calls are tied only to locations and dates, so user preferences expressed in the query or dialogue (e.g., cuisine, room type) never shape tool arguments. Hence we did not reuse TravelPlanner’s sandbox.

Personalised benchmarks: RETAIL and UserBench elicit preferences in dialogue, not from a profile. PTBench, ToolSpectrum and PEToolBench evaluate single tool calls. ETAPP has no tool-choice preferences, includes shorter, less detailed preferences (avg. length 4.67 vs. 17.5 in ToolTailor), and uses LLM-judge scoring. Tau-bench and AppWorld are long-horizon and multi-step, but lack persistent profiles and focus on task completion rather than personalisation.

Our contribution is a benchmark combining multi-step tool use with rich profiles containing tool-choice preferences and fine-grained evaluation of their application. Unlike final-plan evaluation, ToolTailor links preferences to individual tool arguments, allowing us to distinguish failures in preference selection, tool selection, argument generation and planning. We will qualify our novelty claims accordingly.

Scale. Table 2 incorrectly reports the training split, which contains 568 scenarios and 2,313 instances. We will fix the table.

Empirical results

Same-family reference confound. Although Qwen3-14B generated the initial references, they were extensively expert-corrected (per subgoal: 6.9 messages modified, 2.6 added and 0.8 removed on average). The generator itself achieves only 23.5 F1 zero-shot, below Qwen3-4B. Under dependency-aware evaluation, Gemini-3.6-Flash also outperforms Qwen3-8B and Qwen3-14B. Some reference bias may remain, but fine-tuning improves personalised argument prediction across models on unseen user profiles, suggesting that improvements are not attributable solely to reference-style matching. We will moderate the claims in Section 5.

Relevant preferences. Preferences contain no explicit tool labels; their mapping to tools is used during data generation and evaluation. This setting represents an oracle preference retriever, isolating preference application from preference selection. Each LoRA is trained with the corresponding input format, but the model must still infer personalised argument values. We will clarify this distinction and acknowledge that target-independent retrieval remains to be evaluated.

Baselines. All models use identical prompts and text-based output formats, without native function calling. Missing required arguments largely reflect outputs that fail to parse as tool calls (97–99% of cases); among valid calls, fewer than 1% omit required arguments. GPT-4o-mini frequently responds in prose, whereas GPT-5.6-Terra produces empty strings or JSON objects. We will clarify the evaluation setup and its limitations.

Section 5 claims. We will correct the numerical inconsistencies and explicitly report the Calendar and Activities exceptions. We will also clarify that zero-shot models perform better on execution than retrieval, whereas fine-tuning generally improves retrieval but harms execution, consistent with the retrieval-heavy training distribution.

Long-horizon evaluation. Please see our general response on long-horizon evaluation, which reports plan-level results on Test Full. All models struggle to maintain correct personalised decisions over longer trajectories: no model completes a single scenario, and even fine-tuned models typically fail within their first three calls. The error breakdown and data distributions are addressed below.

Data distributions. Test scenarios span 15 themes (1-10% each), profiles average 29 preferences (9 general, 20 tool-specific), and scenarios have 1.2 constraints, split evenly.

Error analysis. Zero-shot models mostly fail by replying in text (16–63% of errors), which fine-tuning removes. The most common error is the wrong next step, i.e. calling a tool for another part of the plan (25-45%). App confusions (5%), search/booking mix-ups (3-10%) and personalised-argument errors (1-15%) are also prevalent. Full statistics and examples will be added to the appendix.

AuthorsOpenReview ↗

Response to Reviewer FXbh (Part 2)

Remaining clarification questions

Q3 Each LoRA uses the input format of its evaluation setting. Preferences contain no explicit tool labels.

Q5 Each segment passes a task-completion checker, followed by expert correction of the trajectory against the scenario constraints, including plan-level validity (Appendix C.2.6).

Q6 “Execution” denotes state-changing functions (booking, calendar). Fine-tuned models mostly fail in the tool and function slots, not arguments: they search instead (40–56% vs. 3–9% zero-shot), skip to the calendar entry, or book another item. Wrong or invented IDs are rare (2–4%). They thus mostly omit the booking call, consistent with retrieval-heavy training data.

Q7 Our heuristic repair leaves some type inconsistencies in the training split, which fine-tuning learns. Most involve Nones in optional arguments. Zero-shot errors are mostly invalid option values and argument names borrowed from other functions. We will add a schema-error breakdown for all models.

Q8 The reported corrections primarily concern crowd annotations, not generation. Experts frequently restored omitted execution subgoals. We acknowledge that the training data remains imperfect; incorporating these findings into automatic repair is a useful extension.

Q9 Each project had 53–62 unique annotators (10.3–12.0 items each on average including controls), so the work was spread across many annotators. Thank you for catching the Table 11 inconsistency: a scenario-matching bug affected the expert statistics, which should read 203 subgoals added over the final 331 test scenarios (0.61 per scenario). Crowd statistics cover 490 annotated scenarios, some of which were later filtered out for generation quality. We will correct Table 11.

Q10 The sizes are 855 (1–2 preferences), 553 (3), and 378 (4); the remaining 159 instances have no relevant preferences.

Thank you Area Chair EYH4 for the reminder, you made me realize my draft was not posted.

Reply to authors

Thank you for the detailed and candid response. I have read both parts. It clarified several points for me, including the training split size, the Table 11 correction, the annotator distribution, and the booking failure pattern. I also appreciate the new plan-level results and error analysis.

I am keeping my score, mainly for three reasons:

  1. Positioning. The planned related-work section and revised Table 1 are welcome, but the contribution relative to prior travel-planning and personalised tool-use work is narrower than the paper currently claims. I would need to see the revised positioning as written.

  2. Comparison with proprietary models. Since all models used a text format without native function calling, and most proprietary-model errors are parsing failures, the comparison still seems to reflect output format more than personalisation.

  3. Relevant-preferences setting. As the subset is derived from the reference tool mapping, I read these gains as an oracle upper bound. A target-independent version would be needed to support the current claim.

The promised changes go in the right direction. I think they are best assessed in a revised version, so I am keeping my current score.

Reviewer_kys3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ToolTailor, a benchmark for evaluating whether language models incorporate user preferences into tool selection and argument prediction in multi-turn travel planning. The authors develop a synthetic data pipeline combining travel scenarios, rich user profiles, subgoals, and tool-use trajectories, with human validation of the evaluation data. The benchmark evaluates next-tool-call prediction given a fixed reference history, measuring overall tool use, personalised arguments, and schema compliance. Dependency-aware matching accommodates valid alternative call orders. Experiments with seven models, including fine-tuning four open models, show improvements from synthetic training while highlighting difficulties in preference identification and substantial differences between retrieval and execution calls.

Critical strengths
  1. The benchmark targets a practically important personalisation capability and provides a useful data resource. User preferences should influence concrete tool choices and arguments, beyond the wording of generated responses. ToolTailor combines rich profiles, multi-turn interactions, and multiple travel tools to support research on this capability. Crowdsourced validation, expert corrections, and incremental review of evaluation trajectories strengthen the practical value of the resource.
  2. The evaluation is detailed and provides useful diagnostic findings. The paper separately measures general tool use, personalised arguments, and schema compliance, compares different preference and tool input settings, and reports both strict and dependency-aware matching. Analyses across subgoals, tool functions, and preference counts, together with results from three inference seeds, provide a substantial empirical picture. In particular, the evaluation reveals contrasting effects of fine-tuning on retrieval and execution calls rather than reporting aggregate improvements alone.
Critical weaknesses
  1. The evaluation provides limited coverage of autonomous long-horizon task performance. Each instance supplies a fixed reference history and independently evaluates the next call. Dependency-aware matching appropriately relaxes ordering constraints, but the protocol does not measure whether models consistently apply preferences, recover from errors, and complete tasks along their own generated trajectories. The authors clearly acknowledge this scope; my concern is therefore about the extent to which the benchmark supports conclusions about complete personalised planning.
Questions for authors
  1. When the target tool and function are provided in both settings, does filtering preferences still substantially improve personalised-argument prediction?
  2. Can rebalancing retrieval and execution training examples reduce execution regressions while preserving the personalisation gains?
4 comments
AuthorsOpenReview ↗

Response to Reviewer kys3

We thank the reviewer for their positive assessment of ToolTailor and for their constructive suggestions. We address the concerns about long-horizon evaluation, preference filtering, and training data balance below.

Long-horizon evaluation

Please see our general response on long-horizon evaluation, where a new adaptive mode evaluates models along their own trajectories, showing that our per-call scores bound performance there and that no current model completes a full personalised plan.

Providing the target tool and function

To isolate the effect of preference filtering on argument generation, we compare full-profile and relevant-preference settings when the correct target tool and function are supplied in both conditions. This removes tool selection as a potential confounding factor and directly tests whether filtering improves the prediction of personalised arguments.

Filtering preferences brings gains of similar size to the all-tools setting (up to +4.6). Much of this gain comes from fewer distractions: with the full profile, irrelevant preferences often lead models to call a tool from another domain instead of the one offered. When models do call the right function, applying preferences remains hard: personalised-argument F1 on these calls is only 45–61% in both settings, with errors in turning preferences into exact values and in deciding when they apply. The challenge is thus both locating the relevant preferences and deciding when and how to apply them, which our argument-level metrics make visible. We will add this analysis to the paper. Zero-shot personalised-argument F1 (%, mean over seeds).

ModelAll tools, all prefsAll tools, relevant prefsTarget tool, all prefsTarget tool, relevant prefs
Qwen3-4B29.131.343.445.1
Qwen3-8B25.327.447.950.9
Qwen3-14B27.029.840.344.6
Gemma3-4B7.68.214.518.4
Gemini-3.6-Flash24.328.742.043.5
GPT-4o-mini12.713.934.238.8
GPT-5.6-Terra24.028.039.739.6

Rebalancing retrieval and execution training examples

We agree that the balance between retrieval and execution examples may contribute to the observed execution regressions after fine-tuning. Our current experiments use the dataset's existing distribution, and we have not systematically investigated alternative sampling ratios.

Our primary contribution is the benchmark and its automatic data-generation pipeline rather than a new training method. Nevertheless, the pipeline makes it possible to generate additional execution-focused examples, making training-data rebalancing a natural extension. We will clarify that the reported regressions reflect the current training distribution and should not be interpreted as an inherent limitation of fine-tuning for personalised tool use.

Thank you for the additional experiments and the thoughtful response. I appreciate the effort to extend the sequential evaluation and investigate the effect of preference filtering.

I have one small clarification regarding the target-tool experiment: are the correct tool and function specified in the prompt while the model remains free to generate the entire call, or are they fixed in the output so that the model only predicts the arguments? The observation that models sometimes call tools from another domain made me wonder whether tool-selection errors still contribute to the reported scores. Clarifying this would help me understand how fully the comparison isolates the effect of preference filtering on argument prediction.

If the tool and function are currently specified only in the prompt, would it be feasible to add results for one or two models using constrained decoding to enforce the correct tool and function, while leaving argument prediction to the model? Comparing the full-profile and filtered-preference conditions under this constraint would help isolate the benefit of filtering for argument prediction.

Thank you again, and have a lovely weekend!

Thank you for the follow-up and your interest in the work! The correct tool and function are specified only in the prompt, so tool-selection errors, such as calls to tools from other domains, do still count in the overall scores.

To isolate argument prediction, we also compared both conditions only on calls where the model used the target function in both settings. There, personalised-argument F1 with relevant preferences is within 2 points of F1 with all preferences. Filtering therefore helps mainly for tool selection, while argument prediction given the correct function is similarly low in both. Using constrained decoding to isolate the effects of tool selection could be an interesting extension of the analysis. Thank you for the suggestion, and have a lovely weekend too!

Thank you for the clarification and the additional analysis. The comparison on calls where both settings select the target function helps clarify that preference filtering primarily benefits tool selection, while personalised argument prediction remains challenging in both conditions. I encourage you to incorporate this distinction into the revised paper.

I will maintain my rating of 3 (Weak acceptance) and remain supportive of acceptance. I appreciate your thoughtful engagement with my questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Long-Horizon Evaluation

To address the concern about evaluating individual tool calls rather than complete trajectories, we conducted a new sequential evaluation on 100 length-stratified Test Full scenarios (1,559 calls, 3–34 per scenario, 15.6 on average).

We compare two evaluation modes:

  • Static: Each prediction uses the reference history, as in the original paper. Previous errors therefore do not affect subsequent predictions.
  • Adaptive: The model's valid actions are incorporated into the history with their recorded tool outputs. A call is accepted if it matches the reference next call or another call whose dependencies are satisfied. Invalid calls are replaced with the reference next call to maintain a valid execution history.

We evaluate function selection (Func), which measures whether the model chooses a valid next tool and function, and exact match (EM), which additionally requires the arguments, including personalised arguments, to be correct.

Results

Table 1. Function-selection accuracy (%)

ModelTrainingStatic strictStatic dep.-awareAdaptive
Qwen3-4BZero-shot37.951.741.3
LoRA40.557.944.1
Qwen3-8BZero-shot44.251.147.2
LoRA36.555.241.8
Qwen3-14BZero-shot41.050.145.1
LoRA41.056.548.5
Gemma3-4BZero-shot9.618.210.7
LoRA32.950.138.7

Table 2. Exact-match accuracy (%)

ModelTrainingStatic strictStatic dep.-awareAdaptive
Qwen3-4BZero-shot15.817.816.6
LoRA13.815.614.8
Qwen3-8BZero-shot19.120.719.7
LoRA12.413.813.2
Qwen3-14BZero-shot15.217.016.1
LoRA14.516.716.4
Gemma3-4BZero-shot2.93.13.0
LoRA10.811.810.8

Results show the mean percentage of correct calls per scenario, averaged over 100 scenarios and three inference seeds.

We draw three conclusions from these experiments.

First, models frequently choose valid alternative execution orders. Depending on the model, 23-39% of accepted function calls differ from the reference next call while satisfying the task dependencies. Thus, adaptive evaluation captures meaningful deviations from reference trajectories rather than simply replaying the original call sequence.

Second, static evaluation provides a useful indication of sequential performance. Across all eight model configurations, adaptive accuracy falls between static strict and dependency-aware accuracy. For example, Qwen3-14B LoRA achieves 41.0% (static strict), 48.5% (adaptive), and 56.5% (static dependency-aware) function accuracy. This suggests that the original per-call evaluation provides informative, albeit imperfect, estimates of performance under sequential execution.

Third, reliable personalised execution remains challenging. Models are considerably better at choosing valid functions than producing fully correct tool calls. Adaptive EM never exceeds 19.7%, and no model completes a scenario without errors. This highlights the difficulty of maintaining accurate personalised decisions across extended interactions.

Overall, these experiments extend ToolTailor's evaluation from independent tool calls to dependency-aware sequential execution, allowing us to assess how models choose valid next steps and apply user preferences along alternative trajectories and highlighting the difficulty of long-horizon personalised tool use.

We acknowledge that this evaluation is not fully on-policy: invalid calls are replaced with reference actions, and tool outputs are prerecorded. Consequently, it does not measure autonomous error recovery or end-to-end task completion. Nevertheless, it provides a more realistic assessment of sequential decision-making and establishes a foundation for fully interactive evaluation. We will include these results and clarify the scope of our claims in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.