acceptodds
Under review as a conference paper at ICLR 2027

ToolRACER: A Robust Agentic Conversation Emulation Resource for Agent Training and Evaluation

21 views

Abstract

Task-oriented conversational agents remain fragile under real world conversation scenarios as they rarely follow a predictable script, especially when users exhibit non-cooperative behavior. Existing function-calling benchmarks often emphasize successful, cooperative interactions and underrepresent adversarial conversation trajectories, thereby limiting the training resources available for developing robust agents. We present ToolRACER, a synthetic data generation pipeline that coordinates a user, assistant and tool agent to generate and validated multi-turn interaction between user and agent. Using ToolRACER, we construct ToolRACER-Bench a robust multi-turn conversation benchmark spanning across six domains, ranging over 55 varied personas, generating a validated corpus of 5.8K conversation trajectories, with approximately 66% of conversations containing failure-prone conversation scenarios. We inject adversarial trajectory behavior, producing validated conversation interaction trajectories that capture realistic, robust scenarios. We evaluate models trained on ToolRACER-Bench against internal benchmark as well as function calling benchmarks such as -bench and ACEBench to evaluate agentic accuracy and agentic robustness. Models trained on ToolRACER-Bench improves end to end agentic accuracy across -bench, BFCLv3 and ACEBench, demonstrating significant gains when mixed with in-domain dataset in small language models for agent capability tasks.

Primary area:
datasets and benchmarks
Keywords:
Function-calling, Synthetic Data generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_F5R3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ToolRACER generates training conversations where users change their requests, provide incomplete information, or encounter tool failures. It uses separate models to simulate the user, assistant, and tools, then checks and repairs the conversations. Training on this data improves some results on external benchmarks, suggesting it can help agents handle difficult situations. However, the paper does not verify whether its simulated users behave like real people, and some evaluation metrics decline after training. Tests with real users and unseen tools would help establish how well these gains carry over to practical use.

Critical strengths
  1. It covers situations where things go wrong. The training data includes users changing their requests, missing information, tool errors, and cases where the agent needs to explain why a task cannot be completed.

  2. Generated conversations are checked and repaired. The pipeline checks tool-call formats, unsupported claims, role confusion, and task outcomes. It also tries to fix conversations that fail these checks.

  3. It tests beyond its own dataset. The authors evaluate on external benchmarks and show improvements on some metrics. They also compare the effects of different types of difficult scenarios.

Critical weaknesses
  1. It does not verify whether the simulated users behave like real people. The models mainly follow predefined personas and behavioral prompts. Their “impatience” or “change of mind” may reflect how AI imagines people behave, rather than how people actually behave.

  2. The model may become better at the authors’ scenarios without becoming broadly better at tool use. The training and internal test data come from the same generation pipeline. External evaluations help, but some gains are small and some metrics get worse, so the evidence for generalization is still limited.

Questions for authors
  1. Human–simulator behavioral alignment. Conduct a controlled study in which human participants and simulated users perform identical tasks under matched conditions. Compare information disclosure, goal revision, acceptance of alternatives, and task abandonment to assess whether simulated behaviors reflect those of real users.

  2. Human evaluation of downstream agents. Evaluate baseline and ToolRACER-trained agents through randomized human–agent interactions, measuring task success, unwanted actions, abandonment rates, and user satisfaction. This would establish whether benchmark improvements translate into better outcomes for real users.

  3. Generalization to unseen tools and scenarios. Construct evaluation splits with disjoint tool sets, scenario templates, and domains, including previously unseen combinations of failures. Evaluate complete interaction trajectories to assess robustness to errors arising during execution.

  4. Compute-matched ablations and statistical reliability. Compare standard and adversity-enriched training data under matched token and optimization budgets. Repeat training and evaluation across multiple random seeds, report confidence intervals, and assess potential trade-offs in standard task performance, unnecessary refusals, and process accuracy.

2 comments

W1

We have provided an example of an unhappy path in Figure 2, illustrating user-oriented adversarial scenarios. Additionally, we have included the prompts and examples of unhappy-path scenario templates used to generate these conversations. Prior to publication, we plan to release the dataset, providing further transparency into how we model user-oriented failures and construct challenging conversational trajectories.

Q3 W2

Both our training and evaluation datasets are generated using the same pipeline and therefore share the same tool schemas. To investigate generalization beyond these schemas, we evaluated our models on established benchmarks, including BFCL v3, τ²-Bench, and ACEBench, which introduce different domains, tool configurations, and interaction settings.

The macro-average results on τ²-Bench, across our main experiments and ablation studies, demonstrate improvements in task success rates compared to the base model and several contemporary baselines.

Another important observation, as reported in Table 3, is that the checkpoint trained on ToolRACERBench-train (3.7k samples) outperforms the checkpoint trained on APIGen-MT (5k samples). Notably, APIGen-MT includes data from the Retail and Airline domains, which overlap with the domains evaluated in τ²-Bench. Despite being trained on fewer samples and without relying on the same degree of domain similarity, our model achieves better performance, highlighting the effectiveness of robust conversational trajectories as training data.

Furthermore, we observe that the checkpoint trained exclusively on Impossible paths (1.2k samples) achieves the best performance in the Hallucination category of BFCL v3 among our training ablations. This finding suggests that exposure to infeasible scenarios during training can improve an agent's ability to recognize and appropriately handle requests that cannot be fulfilled.

Q1 & Q2.

We acknowledge the reviewer's observations regarding human-simulator behavioral alignment and the potential value of evaluating user satisfaction and abandonment rates. While these are important research directions, they fall outside the feasible scope of the present study, given the time and cost they would entail. Our primary objective is not to model human behavior or measure user satisfaction, but rather to generate robust conversational trajectories for training more reliable and capable tool-using agents. While imperfect, we follow the methodology broadly adopted in the field, namely, evaluating resulting models on open-source benchmarks.

Within that methodology, our work is closing a gap in existing agent-training datasets, which often emphasize successful task execution and tool invocation while providing comparatively limited coverage of challenging conversational trajectories. In particular, we aim to investigate how different types of interaction failures influence downstream agent performance. To address this gap, we introduce a taxonomy of conversational trajectories that explicitly characterizes the types of failures being modeled.

An important distinction in our work is that robustness is not considered solely from the perspective of tool invocation or tool execution failures. Instead, we also investigate challenges arising throughout the conversational trajectory, including adversarial user behavior, changing user expectations, and infeasible task requirements. Our objective is to understand whether exposure to these challenging interactions during training improves an agent's ability to handle similar situations during deployment.

To further substantiate our claims, we compare models trained on ToolRACERBench against baselines trained on existing synthetic tool-use datasets, including APIGen-MT, and report comparisons with ToolACE-MT. We evaluate their performance across τ²-Bench, BFCL v3, and ToolRACERBench-test to examine the effectiveness and generalization of our proposed training approach.

Q4.

We plan to extend our experiments across different model families and model sizes to investigate the scalability and generalizability of the improvements observed in this study.

All our current experiments were conducted on the Fireworks platform, which imposed certain constraints on model availability and supported fine-tuning configurations. These limitations restricted our ability to conduct extensive scaling experiments across a broader range of models.

As part of our future work, we also intend to investigate reinforcement learning (RL)-based post-training approaches using robust conversational trajectories. In particular, we aim to explore whether incorporating challenging interaction patterns into RL-based training can further improve agent robustness, adaptability, and task success under adversarial or infeasible conditions.

Thank you for the follow-up and the additional response. I have no further questions.

Reviewer_HZS2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes ToolRACER, a framework for generating synthetic conversations to improve agent robustness. It introduces a corresponding dataset and evaluates its effectiveness through fine-tuning on tool-use benchmarks.

Critical strengths

The paper addresses an important practical problem

Critical weaknesses
  1. Inconsistent references and metric terminology hinder understanding of the paper. Line 263 refers to “Appendix 6,” although the appendices are labeled A–E. Line 412 describes “action accuracy,” whereas Table 4 defines AR as “Action Recall,” without explaining the discrepancy. This ambiguity makes the reported results difficult to interpret.

  2. The pipeline lacks essential reproducibility details. Neither the main text nor the appendices specify the model names and versions for the simulation agents and LLM-based evaluators in Figure 2. Since these models determine trajectory generation and filtering, this omission prevents reliable reproduction of the dataset construction process.

Questions for authors
  1. Do the benefits of ToolRACER-generated training data persist at larger model sizes? The comparison between Qwen3-4B and Qwen2.5-32B changes both model family and parameter count, making the effect of scale difficult to isolate. Do you have evidence from different sizes within the same model family?

  2. Were the initial scenarios and tool definitions manually authored, model-generated, or created through a combination of both? Please clarify the construction process and quantify the human effort and computational cost involved, as these are important for assessing the pipeline’s overall cost and scalability.

1 comment

W1.

Thank you for pointing out this typographical error. To clarify, we adopt the terminology Action Recall (AR) from the referenced paper. In our evaluation, AR measures whether the assistant selects the correct action at each step of the interaction. An action can involve either invoking a tool or generating a natural-language response to the user. Thus, AR evaluates the assistant's ability to select the appropriate action type at each decision point.

Q1.

We plan to extend our experiments in the final version of the paper by evaluating models of varying sizes within the same model family. This will allow us to investigate how effectively the knowledge acquired from our dataset transfers across different model scales and whether the observed performance improvements generalize beyond the current model configuration.

We would also like to acknowledge that all our fine-tuning experiments were conducted on the Fireworks platform, for which we are grateful given limited compute budget. However, as the platform operates independently and supports a specific set of models and fine-tuning configurations, our experiments were subject to certain constraints regarding model availability. (We will document those configurations in detail in the revised paper.)

Consequently, conducting and reporting additional scaling experiments within the current review period may not be feasible. We kindly ask the reviewer to consider these practical limitations. Nevertheless, we recognize the importance of evaluating scalability and intend to incorporate these experiments into the final version of the paper, subject to model availability and computational feasibility.

Q2 W2.

Scenario and Tool Definition Construction. The initial scenario templates and tool definitions were developed through a combination of model-assisted generation and manual review. We further categorized the scenario templates by domain and path type to ensure structured coverage of different interaction settings. The distribution of these templates across domains and path types is provided for additional transparency.

Scenario-Persona Combination and Dataset Splitting. Our dataset is constructed by combining scenario templates with distinct user personas through a cross-product approach. Each scenario-persona combination constitutes a unique generation instance, producing conversations that vary in user behavior, interaction patterns, and tool-use trajectories.

To prevent overlap between the training and test sets at the generation-instance level, each scenario–persona combination is treated as a single, indivisible unit during dataset splitting. Consequently, no identical scenario–persona combination appears in both splits, ensuring disjointness with respect to scenario-persona pairs.

Model and Generation Configuration. The user, assistant, and tool agents all use OpenAI's GPT-5.1 model through the Responses API, with reasoning effort disabled and text.verbosity set to low. Of the four evaluators, three employ the LLM-as-a-judge paradigm using the same GPT-5.1 model, while the Syntax Evaluator is rule-based and deterministic.

For decoding, we set the temperature to 0 for all LLM-based evaluators, while a temperature of 0.6 was used across the three agents during dataset generation. User diversity is introduced through 55 distinct personas, which serve as the primary seeds for the user agent.

Additional Details. We have also classified the initial scenario templates according to their respective domains and path types to provide a clearer overview of the dataset's composition. We will include the above details regarding ToolRACER's scenario construction, generation pipeline, model configurations, dataset partitioning, and evaluation settings in the appendix of the revised paper.

DomainHappyUnhappyImpossibleTotal
Banking10155075
Calendar Assistant15155080
Home Services10204070
Online Shopping10155075
Restaurant Booking10454095
Travel10155075
Total65125280470
Reviewer_LGxWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents ToolRACER, a pipeline for generating and validating synthetic multi-turn data for task-oriented conversational agents. The pipeline coordinates user, assistant, and tool agents and filters trajectories with four automated evaluators that assess syntax, faithfulness, role confusion, and task success. Failed trajectories undergo up to two feedback-guided refinement rounds. The authors use this pipeline to construct ToolRACERBench, which contains 5,643 trajectories (3,762 for training and 1,881 for testing) across six domains, 55 personas, and 72 tools. The data are organized into Happy, Unhappy, and Impossible paths; Unhappy and Impossible cases account for approximately 66.5%.

The main experiments use Qwen3-4B-Instruct as the base model and compare it with APIGen-MT and several open models on ToolRACERBench-Test, BFCLv3, tau2-bench, and ACEBench. The results show several positive signals: training only on ToolRACERBench raises the tau2-bench macro average from 32.20 for the base model to 39.20, while combining it with APIGen-MT yields 41.25. On ACEBench, end-to-end accuracy also rises from 8.4 to 10.0 for the 4B model and from 19.2 to 25.0 for the 32B model. However, process accuracy declines on ACEBench and some BFCL measures also decrease. Together with the absence of repeated runs, human quality-control calibration, and clear evidence of strict train-test isolation, the current results do not yet support the strong claim that the method universally improves robustness without performance costs.

Critical strengths

S1 Structured modeling of realistic failure paths. The paper distinguishes Unhappy paths, in which the task remains feasible but the user is non-cooperative, from Impossible paths caused by tool or policy constraints. It covers goal changes, incomplete information, empty results, timeouts, and policy constraints (Section 3.3, pp. 6-7). Compared with evaluating only cooperative success trajectories, this setting more closely reflects real-world agent deployment and is the paper's strongest conceptual contribution.

S2 A relatively complete engineering pipeline for data generation, filtering, and cross-benchmark evaluation. The four-stage automated validation process and up to two refinement rounds substantially improve the yield of usable data. Appendix A, Table 6 reports that only 26.3% of simulations pass on the first attempt, while refinement recovers 65.4% of the initially failed samples.

Critical weaknesses
  1. Resource validity and independence have not been established. The paper does not clearly identify the models, versions, decoding parameters, random seeds, or degree of independence used for the user, assistant, and tool agents or the four evaluators. It also does not report the scale of human spot checks, annotation criteria, evaluator accuracy, or inter-annotator agreement. ToolRACERBench-Test is generated by the same pipeline, yet the paper only states that the data are divided by domain and path type; it does not establish strict train-test separation at the level of scenario templates, use cases, personas, tool definitions, and seeded entities. The large gains in Table 4 may therefore partly reflect shared generator style or templates rather than robustness to realistic user perturbations. In Table 2, the Entailment rate also falls from 0.071 to 0.042. The authors attribute this to greater variability in realistic user conversations, but provide no human evidence for that interpretation. Resolution criteria: disclose the complete generation and evaluation configuration; conduct a blinded human quality audit and report error types and agreement; provide template-level deduplication and separation statistics; and preferably add an external stress test using independent generators or real conversations.

  2. Experimental controls and claim framing are insufficient . All results are single-point estimates, with no independent training or evaluation repetitions, variance, confidence intervals, or significance tests. Beyond the LoRA configuration, the paper does not report the number of training epochs, maximum sequence length, optimizer, warmup schedule, dataset size after turn splitting, checkpoint-selection procedure, or inference settings. APIGen-MT+ToolRACERBench contains approximately 5K+3.7K trajectories, so its advantage over either dataset alone cannot be separated from the effect of using more training data.

Questions for authors

Q1. Which models and versions were used for the user, assistant, and tool agents and for each of the four evaluators? What were the temperature, top-p, maximum-token, random-seed, and retry settings? Were the generation and evaluation models independent?

Q2. How were the training and test sets split? Were scenario templates, use cases, personas, tool schemas, goal constraints, seeded entities, and near-duplicate conversations isolated across the two sets? Please report overlap rates and the deduplication procedure at each level.

Q3. Was the final dataset subjected to a blinded human audit? Please report the sample size, annotation dimensions, inter-annotator agreement, and the precision/recall or confusion matrices of the four automated evaluators against human judgments. If no such audit was conducted, confidence in the characterization of the corpus as "validated" is substantially reduced.

Q4. How many independent training and evaluation seeds underlie the results in Tables 3-5? Please report means, variances or confidence intervals, and the sampling and termination settings for interactive evaluations such as tau2-bench.

Q5. How do the authors explain the increase in ACEBench end-to-end accuracy alongside the substantial decrease in process accuracy, as well as the decline in BFCL Irrelevance? Please also correct numerical statements such as "52.5% gain" and state clearly which measures do and do not support the claim of "no performance costs."

2 comments

Thank you for your efforts in carefully reviewing this work! Below are the responses to the raised questions:

Q1 W1.

The user, assistant, and tool agents all use OpenAI's GPT-5.1 model through the Responses API, with reasoning effort disabled and text.verbosity set to low. Of the four evaluators, three follow the LLM-as-a-Judge paradigm and use the same GPT-5.1 model, while the Syntax Evaluator is rule-based and deterministic.

For decoding, we set the temperature to 0 for all LLM-based evaluators, whereas a temperature of 0.6 was used across all three agents during dataset generation. User diversity is introduced through 55 distinct personas, which serve as the primary seeds for the user agent.

We will include the parameters of ToolRACER's generation pipeline, model configurations, and evaluation settings in the appendix of the final version of the paper.

Q2.

Our dataset is constructed by combining scenarios with distinct user personas through a cross-product approach. Each scenario–persona combination represents a unique generation instance, resulting in conversations that differ in their user characteristics, interactions, and trajectories.

To prevent data overlap between the training and test sets, each scenario–persona combination is treated as a single, indivisible instance during dataset splitting. Consequently, the same combination cannot appear in both splits.

Although individual scenarios or personas may be reused across different combinations, each scenario–persona pair is unique. This ensures that no identical generation instance is shared between the training and test sets, thereby preventing overlap at the scenario-persona combination level.

Q3 W1.

Conducting human evaluation of the generated dataset would be considerably beyond the scale of the present work, in terms of both time and cost (as it is for most published work in agentic modeling). Instead, we assess the effectiveness of data synthesis through extensive ablation studies and evaluations on established open-source benchmarks, as reported in Tables 3–5.

Additionally, we compared the performance of models fine-tuned on our dataset against checkpoints trained on APIGen-MT, an existing synthetically generated dataset for multi-turn tool use. These comparisons provide empirical evidence of the effectiveness of our generated trajectories in improving downstream agent performance.

Q4.

We report the best-performing checkpoints across the experiments presented in Tables 3-5. During fine-tuning, we explored learning rates ranging from 2e-5 to 1e-4, with LoRA ranks of 16 and 32. The effective batch size was fixed at 32 across all experiments, and the best-performing checkpoint was selected from training runs spanning 1–3 epochs.

All fine-tuning experiments were conducted on the Fireworks platform, and our hyperparameter exploration was constrained to the configuration options supported by the platform.

For evaluations on τ²-Bench and ACEBench, which require a simulated user agent, we used the same GPT-5.1 model, under the benchmarks' default evaluation settings, with a temperature of 0 and a maximum interaction limit of 40 steps.

The configurations and parameters used will be fully documented in the appendix of the revised paper.

Q5.

Our dataset includes conversations with relatively fewer dialogue turns, but potentially longer sequences of tool calls. Such trajectories emphasize sequential and parallel multi-step tool execution, encouraging agents to develop stronger tool-use capabilities within individual interactions.

ACEBench, by contrast, evaluates agent performance across both multi-step and multi-turn settings, introducing a distributional mismatch between our training data and the evaluation tasks.

This mismatch may explain the observed performance differences: models trained on our dataset tend to perform better on multi-step tasks than on multi-turn tasks. Consequently, the stronger multi-step performance can improve end-to-end accuracy (EA), while weaker performance on multi-turn interactions may reduce process accuracy (PA).

Reviewer_LGxWOpenReview ↗

Reviewer Response

Thank you for the detailed responses and clarifications. The additional information regarding model configurations, dataset construction, and training settings is helpful. However, several major concerns remain unresolved. In particular, the lack of human evaluation, incomplete train-test isolation, absence of repeated experiments and statistical analysis, and insufficient clarification of data provenance and ethical considerations still limit the reliability of the proposed approach. Although the authors have provided reasonable explanations for some experimental results, these clarifications do not fully address the fundamental concerns. Therefore, I maintain my original rating of 2 (Weak Reject).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.