- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a reinforcement learning recipe that jointly rewards task intent adherence and behavioral naturalness. The core of the recipe is to operationalize naturalness with a lightweight discriminator trained in an alternating, adversarial loop with the user simulator. The trained model, UserLM-2.0 produces more valid simulated conversations compared wit hteh baseline.
Critical strengths
- The reward design provides a clear idea on what it takes for a user simulator to be good. The idea of operationalizing naturalness with a lightweight discriminator is neat and could be potentially applicable in similar scenarios for shaping the model behavioral.
- The authors compare UserLM-2.0 with many out-of-box LLMs and other approaches for learning a user simulator. UserLM-2.0 demonstrate strong results. The authors further conduct in-depth analysis to provide more insights on why UserLM-2.0 is delivering better outcomes.
Critical weaknesses
- I am unsure whether the evaluation metric will give additional advantage to your method since the reward in RL training specifically optimizes for task fidelity, naturalness, and validity. See my Question 1 below.
- Compared to Human Reference, the Naturalness scores for all the models much lower. The absolute difference is unintuitive to me and I am wondering if there is more insights here.
For other concerns, I think they are minor but please refer to the section below.
Questions for authors
- There are many recent benchmarks for evaluating user simulators. The paper actually surveys them in the Related Work section. I am wondering why you don't evaluate on those benchmarks which seem to be a comparable way for doing the evaluation compared to using you own metrics.
- The Task Fidelity of UserLM-2.0 is still much worse than off-the-shelf LLMs. I am wondering if you see scaling up UserLM-2.0 will continue improving results in different aspects.
4 comments
Response to Reviewer V1KK, Continued
Task-fidelity of UserLM-2.0 and scaling up We would like to push back against the description that “UserLM-2.0 is still much worse”. UserLM-2.0 performs the best on Coverage and second-best on Termination, although it does underperform off-the-shelf LLMs on Quality. We suspect the poorer Quality score may be due to the existence of interactions on WildChat where users repeat past turns verbatim to minimize effort (as seen in the Quality score for Human Reference). UserLM-2.0 is initialized on WildChat and is trained to maintain naturalness against it. The propensity of users to repeat themselves can then conflict with the Quality reward.
The point about scale that the reviewer brings up is interesting! We agree that scaling up the model size will likely improve the task-fidelity of UserLM-2.0. Our focus in this paper was on providing initial experiments for the methodology, and sharing insights about common failure modes one might encounter when using RL for training user simulators. But, broadly studying the scaling laws for user simulators is an exciting direction for future work, as scale might also impact robustness and other facets of the user simulation.
References:
- Xuhui Zhou, Weiwei Sun, Qianou Ma, Yiqing Xie, Jiarui Liu, Weihua Du, Sean Welleck, Yiming Yang, Graham Neubig, Tongshuang Wu, and Maarten Sap. Mind the Sim2real gap in User Simulation for Agentic Tasks. In Third Conference on Language Modeling, 2026.
- Jenna Russell, Marzena Karpinska, and Mohit Iyyer. People Who Frequently Use ChatGPT for Writing Tasks are Accurate and Robust Detectors of AI-generated Text. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
I thank the authors for the detailed response! My questions are clarified a lot and I stay my postive assessment unchanged.
Response to Reviewer V1KK
We thank the reviewer for their response and positive assessment of our paper!
Response to Reviewer V1KK
We thank the reviewer for their positive assessment of our paper! We are similarly excited by future applications of our lightweight discriminator formulation, both with the multi-turn operationalization we consider in the paper as well as with other extensions that can measure turn-level, continuation-level or more abstract behavioral naturalness. We respond to the weaknesses you raised below:
Whether Our Evaluation Advantages UserLM-2.0: This is a good question. We do not believe our evaluation metrics unfairly disadvantage UserLM-2.0. While our training directly optimizes for naturalness, the discriminators we use during evaluation are distinct from those used in train-time. The discriminators in Naturalness-Specific and Naturalness-Generic are both trained from scratch to discriminate against outputs from the exact checkpoints being evaluated, which resets any advantage UserLM-2.0 may have. Pangram, additionally, is completely independent of our training scheme. Both the Quality and Coverage metrics assess general capabilities and we observe prompted assistants, which have not been trained for this, to be performant at both. Finally, all of our evaluations are on a test-set featuring a held-out set of users such that we cannot overfit to the behavior of users seen during training. We also evaluated UserLM-2.0 against a different set of assistants for Reviewers RWyh and cmFd and observed the improvement it has over simulators is robust to the choice of assistant.
We would also like to note that for SFT-trained simulators, each of the desired behaviors (high coverage, proper termination, naturalness) are all demonstrated in the training data. That the SFT models perform poorly on the task fidelity metrics is not because evaluation unfairly advantages UserLM-2.0, but because SFT as a training recipe is insufficient to achieve good performance on task fidelity by itself.
Considering Other Benchmarks: Our focus is on developing user simulators that can generate interactions that are simultaneously naturalistic while adhering to task specifications. To the best of our knowledge, there are no public evaluations which provide such holistic evaluations, which is why we use the construct of validity. While past work such as the User-Sim Index (Zhou et al., 2026) can consider both behavioral naturalness and task evaluations, these are aggregate values that are computed separately. As a result, they do not provide a signal about whether individual interactions are performant on both fronts. Furthermore, as mentioned in the paper, we do not consider human evaluation as past work indicates that human evaluators’ ability to detect LLM text is a function of their own LLM use (Russell et al., 2025). We also do not consider benchmarks such as SimulatorArena due to their use of outdated and deprecated LLMs, and their use of flawed LLM-as-a-judge metrics. Refer to our discussion with Reviewer RWyh for this.
Human Reference Naturalness: We agree that there remains an important gap between human and user simulator performance in terms of naturalness. We attribute this to the diversity of human behavior. SFT can push model behavior to the average user, while RL can further sharpen the model’s distribution. Our use of an iteratively updated discriminator mitigates this, but does not eliminate the problem entirely. As a result, model generated interactions occupy a narrower range of the overall distribution and are more easily identifiable. This is a problem common to any method using SFT or RL, which we do not claim to solve. Methods achieving better quality-diversity trade-offs are necessary to improve user simulators and are an exciting future direction. In short, we believe our evaluation methodology offers a remaining gap that is meaningful to climb: we believe methods that train simulators to cover the gap with human reference naturalness will produce user simulators that improve the state of the art for the community.