The Opponent Learns Too: Alternating Adversarial Training for Stronger Conversational Agents

Conversational agents need to coordinate dialogue and tool use across multiple turns to address evolving user needs. To develop these abilities, agents are trained with simulated users, and recent work has extended this paradigm to jointly train users and agents, yielding interactions that evolve throughout training. However, existing user simulators do not necessarily adapt to the current agent's remaining weaknesses, limiting their ability to provide targeted training signals for further improvement. In this paper, we propose Alternating Training with Opponent Adaptation (ATOA), which uses failures of the current agent to guide successive rounds of adversarial training between the agent and user simulators. Specifically, each round begins by fixing the agent and updating the user simulator using utterances from interactions in which the agent fails. The updated simulator is then fixed to train the agent using successful trajectories together with pairs of successful and failed attempts on the same task. Failures of the newly trained agent are used to guide the next simulator update, allowing the training challenges to evolve across successive rounds. To preserve task consistency, ATOA fixes the user's behavioral decisions in advance and trains only how those decisions are expressed in language, while task completion is independently verified from the resulting environment states. Experiments across different scenarios in ${\tau }^2$-bench show consistent improvements in agent task success.