- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ToolRACER generates training conversations where users change their requests, provide incomplete information, or encounter tool failures. It uses separate models to simulate the user, assistant, and tools, then checks and repairs the conversations. Training on this data improves some results on external benchmarks, suggesting it can help agents handle difficult situations. However, the paper does not verify whether its simulated users behave like real people, and some evaluation metrics decline after training. Tests with real users and unseen tools would help establish how well these gains carry over to practical use.
Critical strengths
-
It covers situations where things go wrong. The training data includes users changing their requests, missing information, tool errors, and cases where the agent needs to explain why a task cannot be completed.
-
Generated conversations are checked and repaired. The pipeline checks tool-call formats, unsupported claims, role confusion, and task outcomes. It also tries to fix conversations that fail these checks.
-
It tests beyond its own dataset. The authors evaluate on external benchmarks and show improvements on some metrics. They also compare the effects of different types of difficult scenarios.
Critical weaknesses
-
It does not verify whether the simulated users behave like real people. The models mainly follow predefined personas and behavioral prompts. Their “impatience” or “change of mind” may reflect how AI imagines people behave, rather than how people actually behave.
-
The model may become better at the authors’ scenarios without becoming broadly better at tool use. The training and internal test data come from the same generation pipeline. External evaluations help, but some gains are small and some metrics get worse, so the evidence for generalization is still limited.
Questions for authors
-
Human–simulator behavioral alignment. Conduct a controlled study in which human participants and simulated users perform identical tasks under matched conditions. Compare information disclosure, goal revision, acceptance of alternatives, and task abandonment to assess whether simulated behaviors reflect those of real users.
-
Human evaluation of downstream agents. Evaluate baseline and ToolRACER-trained agents through randomized human–agent interactions, measuring task success, unwanted actions, abandonment rates, and user satisfaction. This would establish whether benchmark improvements translate into better outcomes for real users.
-
Generalization to unseen tools and scenarios. Construct evaluation splits with disjoint tool sets, scenario templates, and domains, including previously unseen combinations of failures. Evaluate complete interaction trajectories to assess robustness to errors arising during execution.
-
Compute-matched ablations and statistical reliability. Compare standard and adversity-enriched training data under matched token and optimization budgets. Repeat training and evaluation across multiple random seeds, report confidence intervals, and assess potential trade-offs in standard task performance, unnecessary refusals, and process accuracy.
2 comments
Thank you for the follow-up and the additional response. I have no further questions.
W1
We have provided an example of an unhappy path in Figure 2, illustrating user-oriented adversarial scenarios. Additionally, we have included the prompts and examples of unhappy-path scenario templates used to generate these conversations. Prior to publication, we plan to release the dataset, providing further transparency into how we model user-oriented failures and construct challenging conversational trajectories.
Q3 W2
Both our training and evaluation datasets are generated using the same pipeline and therefore share the same tool schemas. To investigate generalization beyond these schemas, we evaluated our models on established benchmarks, including BFCL v3, τ²-Bench, and ACEBench, which introduce different domains, tool configurations, and interaction settings.
The macro-average results on τ²-Bench, across our main experiments and ablation studies, demonstrate improvements in task success rates compared to the base model and several contemporary baselines.
Another important observation, as reported in Table 3, is that the checkpoint trained on ToolRACERBench-train (3.7k samples) outperforms the checkpoint trained on APIGen-MT (5k samples). Notably, APIGen-MT includes data from the Retail and Airline domains, which overlap with the domains evaluated in τ²-Bench. Despite being trained on fewer samples and without relying on the same degree of domain similarity, our model achieves better performance, highlighting the effectiveness of robust conversational trajectories as training data.
Furthermore, we observe that the checkpoint trained exclusively on Impossible paths (1.2k samples) achieves the best performance in the Hallucination category of BFCL v3 among our training ablations. This finding suggests that exposure to infeasible scenarios during training can improve an agent's ability to recognize and appropriately handle requests that cannot be fulfilled.
Q1 & Q2.
We acknowledge the reviewer's observations regarding human-simulator behavioral alignment and the potential value of evaluating user satisfaction and abandonment rates. While these are important research directions, they fall outside the feasible scope of the present study, given the time and cost they would entail. Our primary objective is not to model human behavior or measure user satisfaction, but rather to generate robust conversational trajectories for training more reliable and capable tool-using agents. While imperfect, we follow the methodology broadly adopted in the field, namely, evaluating resulting models on open-source benchmarks.
Within that methodology, our work is closing a gap in existing agent-training datasets, which often emphasize successful task execution and tool invocation while providing comparatively limited coverage of challenging conversational trajectories. In particular, we aim to investigate how different types of interaction failures influence downstream agent performance. To address this gap, we introduce a taxonomy of conversational trajectories that explicitly characterizes the types of failures being modeled.
An important distinction in our work is that robustness is not considered solely from the perspective of tool invocation or tool execution failures. Instead, we also investigate challenges arising throughout the conversational trajectory, including adversarial user behavior, changing user expectations, and infeasible task requirements. Our objective is to understand whether exposure to these challenging interactions during training improves an agent's ability to handle similar situations during deployment.
To further substantiate our claims, we compare models trained on ToolRACERBench against baselines trained on existing synthetic tool-use datasets, including APIGen-MT, and report comparisons with ToolACE-MT. We evaluate their performance across τ²-Bench, BFCL v3, and ToolRACERBench-test to examine the effectiveness and generalization of our proposed training approach.
Q4.
We plan to extend our experiments across different model families and model sizes to investigate the scalability and generalizability of the improvements observed in this study.
All our current experiments were conducted on the Fireworks platform, which imposed certain constraints on model availability and supported fine-tuning configurations. These limitations restricted our ability to conduct extensive scaling experiments across a broader range of models.
As part of our future work, we also intend to investigate reinforcement learning (RL)-based post-training approaches using robust conversational trajectories. In particular, we aim to explore whether incorporating challenging interaction patterns into RL-based training can further improve agent robustness, adaptability, and task success under adversarial or infeasible conditions.