acceptodds
Under review as a conference paper at ICLR 2027

Towards User Simulators That Stay on Task and Still Sound Natural

10 views

Abstract

Users increasingly rely on AI systems for extended, multi-turn tasks, yet evidence indicates that LLM-based system performance degrades over prolonged interaction. To evaluate and improve AI assistants in such settings, we need user simulators that faithfully follow complex user intents while exhibiting realistic conversational behavior. Existing approaches currently manifest a trade-off: prompted assistants can follow complex user intents but lack behavioral fidelity, while specialized user simulators better approximate conversational behavior but struggle to remain faithful to complex intents. In this paper, we propose a reinforcement learning (RL) recipe that jointly rewards task intent adherence and behavioral naturalness. A crucial contribution of the method is operationalizing naturalness with a lightweight discriminator trained in an alternating, adversarial loop with the user simulator. We release the resulting model, UserLM-2.0, which produces valid simulated conversations (i.e., meeting both task-faithfulness and naturalness criteria) 46.2% of the time, over three times the rate of the strongest prior user simulator (14.1%).

Primary area:
foundation or frontier models, including LLMs
Keywords:
user simulation, post-training, naturalness, instruction-following

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_V1KKOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a reinforcement learning recipe that jointly rewards task intent adherence and behavioral naturalness. The core of the recipe is to operationalize naturalness with a lightweight discriminator trained in an alternating, adversarial loop with the user simulator. The trained model, UserLM-2.0 produces more valid simulated conversations compared wit hteh baseline.

Critical strengths
  1. The reward design provides a clear idea on what it takes for a user simulator to be good. The idea of operationalizing naturalness with a lightweight discriminator is neat and could be potentially applicable in similar scenarios for shaping the model behavioral.
  2. The authors compare UserLM-2.0 with many out-of-box LLMs and other approaches for learning a user simulator. UserLM-2.0 demonstrate strong results. The authors further conduct in-depth analysis to provide more insights on why UserLM-2.0 is delivering better outcomes.
Critical weaknesses
  1. I am unsure whether the evaluation metric will give additional advantage to your method since the reward in RL training specifically optimizes for task fidelity, naturalness, and validity. See my Question 1 below.
  2. Compared to Human Reference, the Naturalness scores for all the models much lower. The absolute difference is unintuitive to me and I am wondering if there is more insights here.

For other concerns, I think they are minor but please refer to the section below.

Questions for authors
  1. There are many recent benchmarks for evaluating user simulators. The paper actually surveys them in the Related Work section. I am wondering why you don't evaluate on those benchmarks which seem to be a comparable way for doing the evaluation compared to using you own metrics.
  2. The Task Fidelity of UserLM-2.0 is still much worse than off-the-shelf LLMs. I am wondering if you see scaling up UserLM-2.0 will continue improving results in different aspects.
4 comments
AuthorsOpenReview ↗

Response to Reviewer V1KK

We thank the reviewer for their positive assessment of our paper! We are similarly excited by future applications of our lightweight discriminator formulation, both with the multi-turn operationalization we consider in the paper as well as with other extensions that can measure turn-level, continuation-level or more abstract behavioral naturalness. We respond to the weaknesses you raised below:

Whether Our Evaluation Advantages UserLM-2.0: This is a good question. We do not believe our evaluation metrics unfairly disadvantage UserLM-2.0. While our training directly optimizes for naturalness, the discriminators we use during evaluation are distinct from those used in train-time. The discriminators in Naturalness-Specific and Naturalness-Generic are both trained from scratch to discriminate against outputs from the exact checkpoints being evaluated, which resets any advantage UserLM-2.0 may have. Pangram, additionally, is completely independent of our training scheme. Both the Quality and Coverage metrics assess general capabilities and we observe prompted assistants, which have not been trained for this, to be performant at both. Finally, all of our evaluations are on a test-set featuring a held-out set of users such that we cannot overfit to the behavior of users seen during training. We also evaluated UserLM-2.0 against a different set of assistants for Reviewers RWyh and cmFd and observed the improvement it has over simulators is robust to the choice of assistant.

We would also like to note that for SFT-trained simulators, each of the desired behaviors (high coverage, proper termination, naturalness) are all demonstrated in the training data. That the SFT models perform poorly on the task fidelity metrics is not because evaluation unfairly advantages UserLM-2.0, but because SFT as a training recipe is insufficient to achieve good performance on task fidelity by itself.

Considering Other Benchmarks: Our focus is on developing user simulators that can generate interactions that are simultaneously naturalistic while adhering to task specifications. To the best of our knowledge, there are no public evaluations which provide such holistic evaluations, which is why we use the construct of validity. While past work such as the User-Sim Index (Zhou et al., 2026) can consider both behavioral naturalness and task evaluations, these are aggregate values that are computed separately. As a result, they do not provide a signal about whether individual interactions are performant on both fronts. Furthermore, as mentioned in the paper, we do not consider human evaluation as past work indicates that human evaluators’ ability to detect LLM text is a function of their own LLM use (Russell et al., 2025). We also do not consider benchmarks such as SimulatorArena due to their use of outdated and deprecated LLMs, and their use of flawed LLM-as-a-judge metrics. Refer to our discussion with Reviewer RWyh for this.

Human Reference Naturalness: We agree that there remains an important gap between human and user simulator performance in terms of naturalness. We attribute this to the diversity of human behavior. SFT can push model behavior to the average user, while RL can further sharpen the model’s distribution. Our use of an iteratively updated discriminator mitigates this, but does not eliminate the problem entirely. As a result, model generated interactions occupy a narrower range of the overall distribution and are more easily identifiable. This is a problem common to any method using SFT or RL, which we do not claim to solve. Methods achieving better quality-diversity trade-offs are necessary to improve user simulators and are an exciting future direction. In short, we believe our evaluation methodology offers a remaining gap that is meaningful to climb: we believe methods that train simulators to cover the gap with human reference naturalness will produce user simulators that improve the state of the art for the community.

AuthorsOpenReview ↗

Response to Reviewer V1KK, Continued

Task-fidelity of UserLM-2.0 and scaling up We would like to push back against the description that “UserLM-2.0 is still much worse”. UserLM-2.0 performs the best on Coverage and second-best on Termination, although it does underperform off-the-shelf LLMs on Quality. We suspect the poorer Quality score may be due to the existence of interactions on WildChat where users repeat past turns verbatim to minimize effort (as seen in the Quality score for Human Reference). UserLM-2.0 is initialized on WildChat and is trained to maintain naturalness against it. The propensity of users to repeat themselves can then conflict with the Quality reward.

The point about scale that the reviewer brings up is interesting! We agree that scaling up the model size will likely improve the task-fidelity of UserLM-2.0. Our focus in this paper was on providing initial experiments for the methodology, and sharing insights about common failure modes one might encounter when using RL for training user simulators. But, broadly studying the scaling laws for user simulators is an exciting direction for future work, as scale might also impact robustness and other facets of the user simulation.

References:

  • Xuhui Zhou, Weiwei Sun, Qianou Ma, Yiqing Xie, Jiarui Liu, Weihua Du, Sean Welleck, Yiming Yang, Graham Neubig, Tongshuang Wu, and Maarten Sap. Mind the Sim2real gap in User Simulation for Agentic Tasks. In Third Conference on Language Modeling, 2026.
  • Jenna Russell, Marzena Karpinska, and Mohit Iyyer. People Who Frequently Use ChatGPT for Writing Tasks are Accurate and Robust Detectors of AI-generated Text. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

I thank the authors for the detailed response! My questions are clarified a lot and I stay my postive assessment unchanged.

AuthorsOpenReview ↗

Response to Reviewer V1KK

We thank the reviewer for their response and positive assessment of our paper!

Reviewer_cmFdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a user simulator trained with reinforcement learning to balance task fidelity and conversational naturalness using a dynamically updated discriminator. Comparisons with existing simulators, complexity stress tests, and reward ablations support its effectiveness.

Critical strengths
  1. The paper addresses a clear limitation of existing user simulators: balancing faithful execution of complex intents with natural conversational behavior.
  2. The dynamically updated discriminator provides a practical naturalness reward, while joint optimization addresses common failures in intent coverage, output quality, and termination.
  3. Comparisons with prompted assistants and specialized simulators, together with complexity stress tests and reward ablations, support the effectiveness of the proposed training recipe.
Critical weaknesses
  1. The training recipe is evaluated with a single simulator backbone, assistant model, and data source, leaving its effectiveness across model families, scales, and interaction environments unclear.
  2. Naturalness is assessed primarily through discriminators and AI detectors, without direct human evaluation or downstream experiments demonstrating more reliable assistant evaluation or improved assistant training.
  3. RL improves task fidelity and overall validity but reduces naturalness relative to the model’s own SFT initialization, qualifying the claim that the approach improves task fidelity while preserving natural conversational behavior.
Questions for authors

please refer to the weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer cmFd

We thank the reviewer for their comments. We respond to the weaknesses below:

Experiments with other backbones, datasets or assistants: The reviewer requests experiments with different assistants, different datasets and backbones at different scales. We will respond to these requests one-by-one.

Firstly, with respect to assistants: we ran additional simulations with UserLM-2.0 against three new assistants (GPT-5.6-Luna, GPT-5.6-Terra, Claude Haiku 5.5). The results are summarized as:

User SimulatorAssistantCov.Spec.Gen.Pan.OverallValid %
UserLM-2.0Qwen3.5-9B88.09.436.159.234.946.2
UserLM-2.0GPT-5.6-Luna87.813.337.877.442.851.4
UserLM-2.0GPT-5.6-Terra88.29.937.578.341.952.6
UserLM-2.0Claude Haiku-5.588.58.937.278.541.550.7

In short, UserLM-2.0 achieved conversation validity of 50.7-52.6 against additional assistants, roughly matching the rate against the assistant in our original submission (46.2). This encouraging result indicates that UserLM-2.0 did not overfit to the assistant model it was trained against. To further validate our finding, we also run two additional user simulators against one of these newer assistants (and not all 3 due to cost of running the experiments):

User SimulatorAssistantCov.Spec.Gen.Pan.OverallValid %
UserLM-8BQwen3.5-9B64.212.723.940.213.013.0
UserLM-8BGPT-5.6-Luna63.811.623.959.531.713.9
GPT-5.6-LunaQwen3.5-9B85.80.42.23.62.13.4
GPT-5.6-LunaGPT-5.6-Luna84.90.23.14.82.704.9

We find that both additional simulators we evaluate (UserLM-8B and prompted GPT-5.6-Luna) achieve similar validity rates against the two assistants they are tested in (Qwen-3.5-9B and GPT-5.6-Luna): around 13-14% for UserLM-8B and around 3-5% for GPT-5.6-Luna. This shows that a simulator’s performance in our evaluation setting is not impacted majorly by the choice of the assistant, which is an interesting finding we will add to our Results section. We thank the reviewer for the suggestion.

Secondly, it is not feasible to perform training experiments on different datasets due to a lack of other large collections of Human-AI interaction with a range of LLMs. The closest example we know of is SWE-Chat (Baumann et al., 2026), which would require training a user agent interacting with a terminal environment in tandem with live coding assistants. This requires great infrastructural complexity and it requires significant GPU resources to train. We lack these resources.

Finally, while we agree that scaling up the model size or changing the backbone could give additional insights, our focus in this paper was on providing initial experiments for the methodology, and sharing insights about common failure modes one might encounter when using RL for training user simulators. As each experiment is expensive due to evaluation costs, we prioritized depth in terms of ablation experiments and alternative reward designs rather than testing different models.

AuthorsOpenReview ↗

Response to Reviewer cmFd, Continued

Human Naturalness Evaluations: First, we want to push back against the claim that human-perception of naturalness is the gold standard. It has been established in prior work that human ability to detect LLM-generated text is unreliable (Russell et al., 2025), and this is usually studied on long text excerpts, typically 200+ words, whereas user utterances in human-AI exchanges are typically much shorter (below 30 words).

Second, we view naturalness as a dataset-specific property; natural conversational behavior in a workplace setting is not the same as in a non-professional setting. Human annotators are ill-equipped to grasp these dataset-specific nuances. They would need to be trained to recognize the broad sweep of user behavior in a dataset, which is intractable at worst and highly expensive at best. On the other hand, trained discriminator models are exactly built to evaluate how indistinguishable a sample is from a corpus of real utterances.

We are open to hear what the reviewer thinks, and whether they know of an annotation protocol that would ensure high-quality, reproducible human judgements of naturalness.

Drop in naturalness: The reviewer makes a great observation that on certain naturalness metrics, UserLM-2.0 performs worse than the SFT initialization (UserLM-2.0-SFT). We suspect the drop in naturalness for the Specific and General naturalness discriminators is due to a drop in diversity during RL (Kirk et al., 2024; Yue et al., 2025). We observe RL to sharpen the model’s distribution towards outputs that achieve superior task fidelity. This difference in diversity likely makes outputs from our RL trained model more detectable. We will add a paragraph in the Discussion that connects our finding on the general stagnation of naturalness in our training with this well known RL limitation of loss in diversity due to sharpening of the distribution.

References:

  • Joachim Baumann, Vishakh Padmakumar, Xiang Li, John Yang, Diyi Yang, and Sanmi Koyejo. SWE-Chat: Coding Agent Interactions from Real Users in the Wild. In Third Conference on Language Modeling, 2026
  • Jenna Russell, Marzena Karpinska, and Mohit Iyyer. People Who Frequently Use ChatGPT for Writing Tasks are Accurate and Robust Detectors of AI-generated Text. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
  • Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. "Understanding the Effects of RLHF on LLM Generalisation and Diversity." In The Twelfth International Conference on Learning Representations, 2024.
  • Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? Advances in Neural Information Processing Systems, 2025.
Reviewer_bQkbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the trade-off between task fidelity and behavioral naturalness in user simulation.

  • It introduces UserLM-2.0, which applies multi-turn RL to a user-turn SFT initialization, combining rewards for intent coverage, basic output quality, termination, and naturalness. Its central design is a periodically retrained lightweight discriminator that supplies an evolving real-versus-simulated signal to reduce exploitation of a fixed naturalness reward.
  • Experiments show improved aggregate validity over its own SFT initialization and evaluated prior simulators, with stronger coverage on complex intents. The contribution is primarily a training recipe and empirical findings for user simulation, rather than a new RL or adversarial-learning paradigm.
Critical strengths

1. The problem is clearly stated, and the training objectives address concrete failure modes.

  • The paper studies task fidelity and naturalness together, and addresses repetition, role confusion, and inappropriate termination through its reward design.

2. Targeted ablations support the use of a dynamic discriminator.

  • Comparisons with frozen discriminators, prompted LLM rewards, and reward-removal variants support periodic discriminator updates. The training analyses also show that gains in a fixed reward do not necessarily carry over to independent naturalness scores.

3. The same-backbone SFT baseline provides a useful direct comparison.

  • The comparison helps isolate the effect of the RL stage. Complexity-stratified results and turn-level analyses further show how coverage, information pacing, and repetition change, beyond the aggregate validity score.
Critical weaknesses

1. The naturalness proxies and their practical value need further validation.

  • The ModernBERT discriminators and Pangram provide proxies for textual authenticity, but direct evidence of agreement with human-perceived naturalness is missing.
  • Matching real-user text may reproduce redundancy or disfluency: these behaviors can improve realism without necessarily improving utility for a particular evaluation or training task. The behavioral analyses do not yet establish that balancing naturalness and task fidelity is more useful in practice than prioritizing fidelity.
  • A small blinded assessment of complete dialogues could validate the former gap; the latter claim calls for an application-relevant joint criterion or limited downstream validation, rather than textual-authenticity scores alone.

2. Aggregate validity requires stronger justification and robustness checks.

  • Validity needs a better-grounded evaluation, rather than a valid simulated conversation should be above the bar on both task fidelity and naturalness. It is unclear whether passing these thresholds makes a conversation useful for downstream applications.
Questions for authors

Alternative route with an external harness:

  • Given that some prompted assistants already achieve high task coverage, could a lightweight external harness—such as subgoal tracking, turn-by-turn information release, and output checks—improve their naturalness and performance on complex tasks / stress test while preserving task fidelity? This complements the paper’s route of improving fidelity through RL from an SFT simulator.
  • Please discuss the fidelity–naturalness trade-off this alternative might achieve and the advantages of your training recipe relative to it. If relevant attempts are already available, a limited comparison with the additional inference cost would be helpful.
2 comments
AuthorsOpenReview ↗

Response to Reviewer bQkb

We thank the reviewer for their insightful comments. We respond to the weaknesses below:

Human Naturalness Evaluations: First, we want to push back against the claim that human-perception of naturalness is the gold standard. It has been established in prior work that human ability to detect LLM-generated text is unreliable (Russell et al., 2025), and this is usually studied on long text excerpts, typically 200+ words, whereas user utterances in human-AI exchanges are typically much shorter (below 30 words).

Second, we view naturalness as a dataset-specific property; natural conversational behavior in a workplace setting is not the same as in a non-professional setting. Human annotators are ill-equipped to grasp these dataset-specific nuances. They would need to be trained to recognize the broad sweep of user behavior in a dataset, which is intractable at worst and highly expensive at best. On the other hand, trained discriminator models are exactly built to evaluate how indistinguishable a sample is from a corpus of real utterances.

We are open to hear what the reviewer thinks, and whether they know of an annotation protocol that would ensure high-quality, reproducible human judgements of naturalness.

The Utility of Maintaining Naturalness: We want to clarify that the scope of what naturalness encompasses is not limited to the surface form of the text like disfluency. It also includes numerous other dimensions such as the pace of the conversations, changing intents, rate at which new information is revealed, etc. Prior work (Chang et al., 2025; Laban et al. 2026; Tack et al. 2026) has shown that it is due to these test-time user behaviors, unseen by the LLMs during training, that LLM performance degrades significantly in multi-turn long-horizon settings. Therefore, in response to the reviewer’s larger point, we believe that it is absolutely critical for user simulators to emulate these various behaviors, in addition to task fidelity.

To elaborate, the prior works have argued that the reason for the poor performance of LLM assistants in realistic multi-turn long-horizon tasks is because these test-time user conversations are out-of-distribution with respect to their training data. The goal of user simulators, therefore, is to close this gap between the train and test distributions. The prompted user simulators have high task fidelity but do not align with real user conversational behaviors. Therefore, they would fail to eliminate the above gap and assistants trained against them would similarly not be robust to these behavior dynamics. In fact, recent work (Suh et al., 2026) trained LLM assistants against both prompted assistants (high task fidelity) and trained simulators (higher naturalness) and showed that assistants trained against the latter perform better in realistic settings. This provides further evidence that optimizing naturalness of user simulators is critical for downstream assistant training.

Justification for Validity: Our validity metric is primarily intended to provide a holistic measure for an individual interaction. Prior work on evaluation predominantly computes measures of naturalness and task performance independently (Zhou et al., 2026). This makes it difficult to disentangle whether interactions that perform strongly on task fidelity, for instance, also perform well on naturalness. Given that our goal is to train a model that simultaneously achieves strong task fidelity and naturalness, a joint measure of these two capabilities is necessary. Validity serves this exact need by requiring both task fidelity and naturalness to pass a baseline performance. Finally, we would like to refer to our earlier argument for why we believe that both naturalness and task fidelity are crucial components of a good simulation.

AuthorsOpenReview ↗

Response to Reviewer bQkb, Continued

Alternative Harness: Developing an external harness to make off-the-shelf LLMs better simulators is an interesting, if orthogonal, direction. However, we believe that making such a harness succeed would be difficult due to two reasons. Firstly, the harness assumes access to subgoals. The exact subgoals are oracle information that is provided to the coverage judge alone. This would unfairly advantage the harness. Secondly and more importantly, the problem of naturalness still remains. Prompting alone cannot capture the full diversity of user behavior. This is the case for both the distribution of language in a given turn, but also the distribution of how users reveal subgoals over time. If our understanding is correct, the reviewer’s suggested external harness would have hard-coded rules for when and how frequently to reveal new subgoals. However, such hardcoded rules would not be grounded in real data or cover only a subsection of it at best. For such a harness to succeed, we would need to extract specialized prompts targeting different clusters of behavior both in language and conversation dynamics and dynamically prompt models with these clusters in a way that respects the true distribution.

To perform this experiment properly is non-trivial and would be a research project in its own right.

References:

  • Jenna Russell, Marzena Karpinska, and Mohit Iyyer. People Who Frequently Use ChatGPT for Writing Tasks are Accurate and Robust Detectors of AI-generated Text. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025.
  • Serina Chang, Ashton Anderson, and Jake M. Hofman. "Chatbench: From Static Benchmarks to Human-AI Evaluation." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.
  • Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. "LLMs Get Lost in Multi-Turn Conversation." International Conference on Learning Representations, 2026. 2026.
  • Jihoon Tack, Philippe Laban, and Jennifer Neville. "LLMs Get Lost in Evolving User Intent." arXiv preprint arXiv:2607.20734 (2026).
  • Joseph Suh, Ayush Raj, Minwoo Kang, and Serina Chang. "Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants." arXiv preprint arXiv:2605.09808 (2026).
  • Xuhui Zhou, Weiwei Sun, Qianou Ma, Yiqing Xie, Jiarui Liu, Weihua Du, Sean Welleck, Yiming Yang, Graham Neubig, Tongshuang Wu, and Maarten Sap. Mind the Sim2real gap in User Simulation for Agentic Tasks. In Third Conference on Language Modeling, 2026.
Reviewer_RWyhOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces UserLM-2.0, a multi-turn user simulator trained with RL to improve task fidelity while retaining realistic conversational behavior. The method combines subgoal coverage and termination rewards with a naturalness reward from a periodically retrained discriminator. On the authors’ held-out WildChat evaluation, validity improves relative to the same SFT checkpoint. Ablations show that frozen discriminators and LLM-judge naturalness rewards get reward-hacked.

Critical strengths
  1. The problem matters, and splitting simulator quality into task fidelity and naturalness is a clear framing.
  2. The recipe is carefully engineered. The negative results regarding frozen discriminators and LLM judges leading to longer utterances and reward hacking, are interesting.
  3. The comparison is broad. The paper also includes fine-grained behavioral metrics and an analysis of training dynamics.
  4. The artifact is public, which is valuable to the community.
Critical weaknesses
  1. The method’s broader insight is unclear. The authors present the RL recipe as the main contribution, but the paper never states what general lesson it offers. Jointly training a dialogue generator and discriminator has been done before (Li et al., 2017, which the paper cites). The recipe also reuses existing parts: flipped-dialogue SFT, GRPO, and LLM-judge rewards. As written, the method is a list of components with no formal objective, so it is hard to say what is new beyond applying a known approach to user simulation.
  2. Evaluation is tied to the training setup. Most results use the same assistant as RL training (Qwen3.5-9B), and the Quality metric uses the same judge as the Quality reward. The simulator may have adapted to this particular assistant, so it is unclear whether the gains hold with other assistants.
  3. Naturalness drops after RL. Compared with UserLM-2.0-SFT, UserLM-2.0 scores lower on both trained-discriminator naturalness measures and on overall naturalness; only Pangram improves (Table 1). This conflicts with the claim of improving fidelity “without sacrificing naturalness” (L88, L124). RL also shifts more sub-goals into the first turn, further from human behavior than the SFT model (Table 6). This looks like the reward hack Sec. 2.1 warns about.
  4. Closely related work is missing. The related-work section does not discuss: DIAL (Zhu et al., arXiv:2512.20773), which adversarially trains LLM user simulators against iteratively retrained discriminators; Keizer et al. (arXiv:2306.00858), which adversarially trains neural user simulators for dialogue policy optimization.
Questions for authors
  1. Is the contribution mainly an application of adversarial dialogue training to user simulators? If not, what is the key new idea? A formulation section would help, stating: which objective the combined reward optimizes (e.g., matching the distribution of real conversations for a given intent, subject to coverage); how it relates to GAN or adversarial imitation objectives; why the rewards are combined as they are (quality multiplied in as a gate, the other terms added).
  2. Can you report results on a public benchmark such as SimulatorArena (Dou et al., 2025)? For example: do assistant ratings produced with UserLM-2.0 correlate better with human ratings than ratings produced with baseline simulators? If not, please explain why. At a minimum, please evaluate with held-out assistants from other model families.
  3. How do you interpret the drop in naturalness after RL? Does RL trade naturalness for coverage, so that the higher validity mainly comes from the lenient rule of passing any one of three discriminators? Please provide explanation or related evidence.
  4. Please discuss additional related works and state how your method differs, e.g., online multi-turn GRPO versus offline DPO, adding task-fidelity rewards, and the open-domain setting.
5 comments
AuthorsOpenReview ↗

Response to Reviewer RWyh

We thank the reviewer for their comments. We address the weaknesses below:

The method’s broader insight is unclear: We clarify our contribution succinctly. We address a key limitation of existing approaches to user simulation, which are instantiated either with prompted off-the-shelf LLMs, which are strong at instruction following but are unnatural, or with finetuned user simulators, which approximate the user distribution but fail at instruction following. Our training recipe with rewards for task fidelity and naturalness produce models that are strong at both fronts and therefore address this flaw.

As the reviewer states, prior works introduce the idea of “flipping the dialogue” SFT and GRPO, and our work builds on that line of work. However, naively combining these prior components (SFT w/ flipped dialogue + GRPO w/ LLM-judge rewards for naturalness) fails to yield good user simulators. Specifically, our key insight with the naturalness reward is that the conventional use of LLM judges to operationalize naturalness (seen both in evaluation benchmarks like SimulatorArena (Dao et al., 2025) or post-training recipes (Wu et al., 2026; Wang et al., 2026)) is prone to severe reward hacking and is therefore a poor proxy for naturalness. Adversarial training with discriminators is a strong alternative to this existing norm. This is both explicitly stated in Sections 2.2 and 2.3 of our Method section and empirically shown in Section 4.3. The reviewer themselves notes these insights as a strength of the paper. We would also like to note that Li et al, 2017 is concerned with arbitrary dialogue generation using OpenSubtitles. This has no instruction following components and is an entirely different task setting (i.e., not the human-AI simulation setting we focus on).

Finally, we would like to note that several frontier labs and published academic papers re-use the shared high-level backbone of warm-start SFT followed by RL with GRPO and LLM-judge rewards (DeepSeek-AI, 2026; GLM-5 Team, 2026). Under the reviewer’s criteria, any paper operating within this shared paradigm would lack novelty. Our contribution is not in the mechanistic steps we apply, but in novel reward design for a specific task, and empirical demonstration of the effectiveness of the method to achieve a well-formulated objective: training user simulators that can achieve high task fidelity while not sacrificing naturalness of interaction.

Evaluation scheme: We appreciate the reviewer’s insight that we might have overfit to the assistant we trained against, and agree that evaluating simulators against multiple assistants would be a way to evaluate generalization and robustness of the simulator. To that end, we ran additional simulations with UserLM-2.0 against three new assistants (GPT-5.6-Luna, GPT-5.6-Terra, Claude Haiku 5.5). The results are summarized as follows:

SimulatorAssistantCov.Spec.Gen.Pan.OverallValid %
UserLM-2.0Qwen3.5-9B88.09.436.159.234.946.2
UserLM-2.0GPT-5.6-Luna87.813.337.877.442.851.4
UserLM-2.0GPT-5.6-Terra88.29.937.578.341.952.6
UserLM-2.0Claude Haiku-5.588.58.937.278.541.550.7

In short, UserLM-2.0 achieved conversation validity of 50.7-52.6 against additional assistants, roughly matching the rate against the assistant in our original submission (46.2). This encouraging result indicates that UserLM-2.0 did not overfit to the assistant model it was trained against. To further validate our finding, we also run two additional user simulators against one of these newer assistants (and not all 3 due to cost of running the experiments):

SimulatorAssistantCov.Spec.Gen.Pan.OverallValid %
UserLM-8BQwen3.5-9B64.212.723.940.213.013.0
UserLM-8BGPT-5.6-Luna63.811.623.959.531.713.9
GPT-5.6-LunaQwen3.5-9B85.80.42.23.62.13.4
GPT-5.6-LunaGPT-5.6-Luna84.90.23.14.82.704.9

We find that both additional simulators we evaluate (UserLM-8B and prompted GPT-5.6-Luna) achieve similar validity rates against the two assistants they are tested in (Qwen-3.5-9B and GPT-5.6-Luna): around 13-14% for UserLM-8B and around 3-5% for GPT-5.6-Luna. This shows that a simulator’s performance in our evaluation setting is not impacted majorly by the choice of the assistant, which is an interesting finding we will add to our Results section. We thank the reviewer for the suggestion.

AuthorsOpenReview ↗

Response to Reviewer RWyh, Continued

Regarding the idea to evaluate under the Simulator Arena paradigm: We attempted to perform this additional analysis, but unfortunately ran into logistical issues that render this evaluation impossible. Several of the assistants required for evaluation (e.g., Claude Sonnet 3.5 and GPT-4-Turbo) are now deprecated. As the original Simulator Arena was validated against these specific assistants, we would need to meta-evaluate the Arena with new assistants before it can be reliably used for evaluation.

Further, part of the evaluation protocol requires LLM-as-a-judge prompting to evaluate high level concepts such as naturalness and “interaction quality”. As our experiments have shown and is now known in the community, LLM-as-a-judge based evaluation for these concepts is not valid and prone to reward hacking. Because of the operational difficulty and the limitations around LLM-as-a-judge, we did not pursue this direction further, but we would be happy to hear if there’s a scoped experiment the reviewer had in mind.

Drop in naturalness: The reviewer makes a great observation that on certain naturalness metrics, UserLM-2.0 performs worse than the SFT initialization (UserLM-2.0-SFT). We suspect the drop in naturalness for the Specific and General naturalness discriminators is due to a drop in diversity during RL (Kirk et al., 2024; Yue et al., 2025). We observe RL to sharpen the model’s distribution towards outputs that achieve superior task fidelity. This difference in diversity likely makes outputs from our RL trained model more detectable. We will add a paragraph in the Discussion that connects our finding on the general stagnation of naturalness in our training with this well known RL limitation of loss in diversity due to sharpening of the distribution.

With respect to the reviewer’s comment on Table 6 of the Appendix: This is not evidence of reward hacking. The Human Reference achieves a coverage score of 49% on the first turn. UserLM-2.0 overshoots this with a score of 59%, while UserLM-2.0-SFT, being poor at instruction following, undershoots this with a score of 44.3%. On the other behavioral metrics we have, on the other hand, UserLM-2.0 is much closer to the Human Reference. For the Revisitation and Repeat-only metrics, which investigate to what extent models repeat previously covered subgoals, UserLM-2.0 achieves 39.1% and 33.0% while the Human Reference achieves 38.8% and 33.3%. This is a near exact replication. RL training also reduces the number of turns spent without addressing any subgoal. For the Human Reference, this is 6.8%. UserLM-2.0 is very close to this score at 4.4%, while UserLM-2.0-SFT wastes 18.7% of its turns.

Related work: We thank the reviewer for bringing these papers to our attention. We will incorporate them into our related work. The closest to our work is DIAL. While they do use adversarial training, their primary goal is the production of more naturalistic interactions, without any rewards for task fidelity. However, these two can often provide conflicting signals, as naturalness can be easy to optimize by sacrificing task fidelity. Our motivation is to address this same gap in existing user simulators, which satisfy task fidelity or naturalness at the cost of the other. Our training pipeline with rewards for both task fidelity and naturalness addresses this gap. Our alternative reward experiments and our ablation without rewards further empirically demonstrate that prevalent approaches for instantiating naturalness are prone to reward hacking and harm naturalness. DIAL has no such experiment. The Keizer et al, 2023 paper is more distantly related as it deals with spoken dialogue systems.

AuthorsOpenReview ↗

References

References:

  • Yao Dou, Michel Galley, Baolin Peng, Chris Kedzie, Weixin Cai, Alan Ritter, Chris Quirk, Wei Xu, and Jianfeng Gao. "SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants?." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
  • Shirley Wu, Evelyn Choi, Arpandeep Khatua, Zhanghan Wang, Joy He-Yueya, Tharindu Cyril Weerasooriya, Wei Wei, Diyi Yang, Jure Leskovec, and James Zou. HumanLM: Simulating Users with State Alignment Beats Response Imitation. In Forty-third International Conference on Machine Learning, 2026.
  • Yingshan Susan Wang, Cedegao E. Zhang, Linlu Qiu, Zexue He, Pengyuan Li, Alex Pentland, Roger Levy, and Yoon Kim. "Learning User Simulators with Turing Rewards." In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pp. 31797-31844. 2026.
  • Anyi Xu, et al. "Deepseek-v4: Towards Highly Efficient Million-token Context Intelligence." arXiv preprint arXiv:2606.19348 (2026).
  • Aohan Zeng, et al. "Glm-5: from Vibe Coding to Agentic Engineering." arXiv preprint arXiv:2602.15763 (2026).
  • Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, Roberta Raileanu. "Understanding the Effects of RLHF on LLM Generalisation and Diversity." ICLR 2024.
  • Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? Advances in Neural Information Processing Systems, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.