- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a training approach called ATOA, which is an iterative process of training an AI conversational agent and a user simulator. The user simulator learns from failures on tasks where the conversational agent sometimes succeeds. The conversational agent then uses this revised user model as its training partner using RAFT on successful conversations, followed by DPO on pairs of successful and failed conversations from the same task. One of the main design choices was to maintain a fixed controller for all user decisions. Only the user model can learn how to communicate these decisions. The goal of this design choice is to create challenges that reflect the conversational agent's shortcomings while maintaining a consistent environment for the task. The authors evaluated ATOA on the Retail, Airline, and Telecom tasks developed from τ²-bench at two different model sizes.
In addition to comparing ATOA with other baseline approaches, the authors conducted component ablations, continuation experiments with controls, and evaluations with two other user models. The results indicate that ATOA improved performance under the authors' scripted-user evaluation, particularly in the Retail and Telecom domains. However, with the standard GPT-4.1 user it fell below the untrained base agent in Retail and Airline, while improving performance in Telecom.
Critical strengths
The paper contains multiple solid experimental controls. The authors start from the same agent checkpoint and then compare continuing to train on the previous user to training on the updated user, and they include a budget-matched version of this comparison that uses the same number of preference pairs and DPO steps. They also perform component-level ablations. All agents are tested using the same user and using the same behavioral scripts, and success is defined based on the state of the environment. The paper shows results where performance decreases, and there are useful implementation details in the appendix and supplementary code to allow one to analyze the most important aspects of the simulator. Thus, differences in the main results table can be attributed to the trained agents rather than to the evaluation user, and the shortcomings described below can be traced back to particular aspects of the experimental protocol.
Critical weaknesses
Evidence that ATOA generalizes to new users is insufficient. Most of the evaluations of ATOA utilize the author's own scripted-user framework and behavioral controller, and the user in the evaluation shares the same behavioral controller and the same script-trained language renderer with ATOA's first training opponent. Therefore, it cannot be ruled out that ATOA has learned the interaction protocol of this opponent. ATOA performance relative to the base agent using the standard GPT-4.1 user is worse by 5.35 and 5.25 percentage points on Retail and Airline respectively. Additionally, ATOA performs worse than fixed-user training. On the other hand, for Telecom, there is a large improvement of 27.50 percentage points. The GPT-5.5 renderer experiment utilized the same behavioral rules and GPT-5.5 was also the teacher model used to create the user SFT data. Due to this fact, the purpose of this experiment was primarily to test robustness to alternative language realizations while maintaining an identical behavioral policy for the user. As a result, the experimental results do not demonstrate that ATOA reduces opponent-specific gains in general. The main methods utilized in this study should be tested using independent user models and behavioral policies. In addition, the performance analyses for Retail and Airline need to be explained.
Questions for authors
-
Could you evaluate Base, Fixed-user, and ATOA using a standard GPT-4.1 User and a User Simulator from a different model family (with matched tasks and budgets) and provide an explanation for why ATOA performed less well than the Base Agent in Retail & Airline under evaluation by the GPT-4.1 User.
-
Could you evaluate whether the improvements made to ATOA were transferable to Users who followed differing behavioral policies or interaction scripts rather than just having their language rendered differently (i.e., using a different controller) so that we can determine which improvements are generalizable and which are specific to the interaction protocol.
-
Could you report the success rates of the Post-RAFT and Post-DPO agents across training rounds to illustrate how much of the improvement can be attributed to RAFT vs. DPO.
4 comments
Response to Reviewer mSEE (2/2)
Q3. Contributions of RAFT and DPO across training rounds
We thank the reviewer for this helpful suggestion. While the RAFT Only and DPO Only ablations in Table 2 demonstrate the importance of both components, we agree that reporting intermediate checkpoints would provide a clearer view of their contributions during alternating training.
Additional stage-wise evaluation. We evaluate the Post-RAFT and Post-DPO checkpoints after each training round of Qwen3.6-27B, using the same held-out tasks, scripted evaluation user, and trial seeds as our main experiments. We report the gain from RAFT relative to the preceding round's agent, followed by the additional gain from DPO.
Table R3. Stage-wise task success across ATOA rounds
| Domain | Round | Post-RAFT (%) | Post-DPO (%) | RAFT gain (pp) | DPO gain (pp) |
|---|---|---|---|---|---|
| Retail | R1 | 32.96 | 34.21 | +7.64 | +1.25 |
| R2 | 38.72 | 40.38 | +4.51 | +1.66 | |
| R3 | 42.91 | 44.85 | +2.53 | +1.94 | |
| R4 | 46.31 | 47.84 | +1.46 | +1.53 | |
| Airline | R1 | 7.35 | 7.86 | +0.92 | +0.51 |
| R2 | 8.66 | 9.16 | +0.80 | +0.50 | |
| R3 | 9.51 | 9.74 | +0.35 | +0.23 | |
| Telecom | R1 | 77.63 | 79.88 | +5.37 | +2.25 |
| R2 | 82.96 | 84.42 | +3.08 | +1.46 | |
| R3 | 84.92 | 85.76 | +0.50 | +0.84 |
The stage-wise comparison shows that RAFT provides substantial improvements in the early rounds by learning from verified successful trajectories, while DPO contributes additional gains by contrasting successful and failed attempts on the same tasks. As training progresses, the gains from RAFT become smaller, whereas DPO continues to provide complementary improvements, particularly in the later Retail rounds.
W1. Clarifying the scope of cross-user generalization
We appreciate the reviewer's careful observations regarding our evaluation protocol. The GPT-5.5 experiment in Table 15 evaluates robustness to different language realizations under the same behavioral controller. Since GPT-5.5 also serves as the teacher for our initial user-SFT data, this experiment establishes language-level robustness rather than generalization to independent behavioral policies. Our independent-user and alternative-controller evaluations in Q1 and Q2 directly examine the latter.
Importantly, Fixed-user training and ATOA differ in more than opponent adaptation (Table 16). Fixed-user training uses an untrained user language renderer, whereas ATOA first applies script-based SFT and subsequently performs failure-guided opponent adaptation. Therefore, their performance difference cannot be attributed solely to opponent adaptation. To isolate this contribution, our Fixed Opponent ablation (Table 2) retains the same script-based user initialization but disables subsequent opponent updates. Under the scripted-user evaluation, ATOA improves over Fixed Opponent by 5.31 points on Retail, 1.58 points on Airline, and 7.85 points on Telecom. This controlled comparison directly demonstrates the additional benefit of opponent adaptation beyond user initialization.
The cooperative-user results highlight an important trade-off between robustness to challenging interactions and performance under highly cooperative behavior. ATOA is designed to strengthen agents against users who disclose information gradually, change requests, or exhibit limited patience. Under our challenging scripted-user evaluation, ATOA improves over Base by 22.52 points on Retail and 13.50 points on Telecom. Under the cooperative GPT-4.1 user, it retains success rates of 75.50% on Retail and 62.04% on Airline, despite decreases of 5.35 and 5.25 points relative to Base, while achieving a substantial 27.50-point improvement on Telecom. These results demonstrate that the benefits of challenging-user training are not uniform across interaction policies: ATOA delivers substantial gains under demanding interactions while maintaining considerable effectiveness with cooperative users, with the balance varying by domain.
Taken together, our controlled ablations establish the additional contribution of failure-guided opponent adaptation, while the independent-user and alternative-controller evaluations examine how these improvements transfer beyond the original training interaction protocol.
We sincerely thank the reviewer for the constructive feedback and valuable suggestions. We hope our clarifications and additional experiments address the main concerns and provide stronger evidence for ATOA's effectiveness and generalization. If our responses have adequately addressed your concerns, we would greatly appreciate your reconsideration of the rating.
Thanks for your response. Many of my concerns have been addressed. I will keep my score.
Thank You for Your Careful Review and Engagement in the Discussion
Thank you for your follow-up and for acknowledging that many of your concerns have been addressed. We appreciate your careful consideration of our responses and additional evidence.
If there are any remaining concerns that are particularly important to your overall assessment, we would be very happy to provide further clarification while the discussion is still open. We hope our additional experiments and revisions can be taken into account in your final evaluation.
Thank you again for your time and thoughtful feedback.
Response to Reviewer mSEE (1/2)
We thank the reviewer for the careful assessment and constructive suggestions. We especially appreciate the recognition of our controlled experiments and ablation studies. Below, we address the concerns regarding cross-user generalization, behavioral-policy transfer, and the contributions of RAFT and DPO.
We agree that our GPT-5.5 renderer experiment primarily examines robustness to alternative language realizations, rather than generalization to independent user policies. To address this limitation, we extend our evaluation to a user simulator from a different model family.
Additional evaluation with an independent user simulator. We evaluate Base, Fixed-user, and ATOA using both the standard GPT-4.1 user and a Llama-3.3-70B-based user simulator. Unlike our training opponent, the Llama user independently determines its responses and interaction decisions from the task goal and relevant private information, without accessing our behavioral controller, interaction scripts, or script-trained language renderer.
For direct comparability with Table 16, we retain the original evaluation tasks and budgets: 67 Retail tasks with six trials each, 27 Airline tasks with four trials each, and 40 Telecom tasks with two trials each. All agents use the same checkpoints, tools, and environment-based task verifiers.
Table R1. Cross-user evaluation with independent simulators
The Llama evaluation tests whether ATOA's gains extend beyond the original scripted protocol. ATOA outperforms both baselines across all three domains, while the GPT-4.1 results reveal policy-dependent generalization.
Under cooperative GPT-4.1 evaluation, ATOA trails Base by 5.35 and 5.25 points on Retail and Airline, but improves Telecom by 27.50 points. This reflects a trade-off between training for challenging interactions, including delayed disclosure and limited patience, and performance with highly cooperative users.
ATOA aims to improve robustness to challenging interactions, not guarantee gains under every user policy. The independent-user results provide further evidence that its benefits extend beyond the training opponent.
We agree that changing the language renderer alone does not establish generalization to different user behaviors. To examine this more directly, we introduce an independently implemented behavioral controller with interaction policies that differ from those used during ATOA training.
Additional cross-policy evaluation. We consider two user policies: a Proactive user who readily provides relevant information and cooperates with clarification requests, and a Delayed-disclosure user who provides information more gradually and requires additional clarification. Unlike our training controller, these policies independently determine information-disclosure timing and dialogue decisions. Both preserve the original task goals and constraints, and use the same Llama-3.3-70B language renderer to express their decisions.
We evaluate the frozen Base, Fixed-user, and ATOA agents on the same held-out tasks and trial budgets used in Table 16, keeping the agent tools and environment verifiers unchanged.
Table R2. Generalization to alternative behavioral policies
The comparison suggests that ATOA's benefits can extend beyond the original behavioral controller. Under the independently defined delayed-disclosure policy, ATOA outperforms both baselines across all three domains. Importantly, it also maintains competitive performance under the proactive policy, matching or slightly exceeding Base on Retail, trailing by only 0.92 percentage points on Airline, and retaining a substantial advantage on Telecom.
This result would indicate that ATOA's improvements are not confined to its training interaction protocol, while the magnitude of the gains remains dependent on user behavior and task domain.