acceptodds
Under review as a conference paper at ICLR 2027

Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants

Abstract

User simulators are increasingly leveraged to build interactive AI assistants, yet how to measure the quality of these simulators remains an open question. In this work, we show how simulator quality can be quantified in terms of its downstream utility: how an LLM assistant trained with this user simulator performs in the wild when interacting with real humans. In a controlled experiment where only the user simulator varies, we train LLM assistants via reinforcement learning against a spectrum of simulators, from an LLM prompted to role-play a user to one fine-tuned on human utterances from WildChat. As evaluation, we measure pairwise win rates in a user study with 283 participants and on WildBench, a benchmark derived from real human–AI conversations. Training against the role-playing LLM yields an assistant statistically indistinguishable from the initial assistant in our user study (51% win rate), whereas training against the fine-tuned simulator yields significant gains (58% over the initial and 57% over the one trained against role-playing). Closer inspection reveals three further patterns: methods for making role-playing LLMs more realistic (e.g., persona conditioning) improve trained assistants but do not close the gap to the fine-tuned simulator; scaling the simulator's model size benefits the fine-tuned simulator but yields marginal gains for role-playing ones; and assistants trained against role-playing simulators fail to generalize when paired with other simulators at test time, while the one trained against fine-tuned simulator does. Together, these results argue for grounding user simulators in real human behavior and measuring their quality by their downstream effect on real users.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Human interaction in AI systems, Language models, Dialogue simulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_C9NwOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks a practical question: does training with simulated users actually make assistants better for real people? It compares role-playing, persona-conditioned, and human-data-fine-tuned simulators through controlled training experiments, automated evaluations, and a study with 283 participants. The main finding is that improvements during simulated training do not necessarily transfer to human interactions, while training with a fine-tuned user simulator produces measurable gains in human preference.

Critical strengths
  1. The paper evaluates what ultimately matters. It measures simulator quality through its impact on trained assistants and real users, rather than relying solely on how realistic simulated conversations appear.

  2. The comparisons are carefully controlled. Keeping the assistant, training algorithm, and reward setup consistent makes the effects of simulator choice easier to assess. Cross-simulator testing adds useful evidence about transfer.

  3. The human study provides meaningful validation. The fine-tuned-simulator-trained assistant achieves a 58.1% preference win rate against the initial assistant, while ordinary role-playing training shows no significant advantage. Reporting both positive and negative findings makes the study informative for practitioners.

Critical weaknesses
  1. The human study misses two important baselines. Neither the persona-conditioned variant nor STATIC is included. STATIC trains directly on real conversation histories and performs similarly to the fine-tuned simulator approach on WildBench, leaving the additional value of multi-turn simulation insufficiently established.

  2. The persona baseline is limited. A small set of behavioral prompts does not represent the full potential of persona-based simulation. Since the fine-tuned simulator also receives substantial human supervision, the results do not fully separate the benefits of fine-tuning from access to human behavioral data.

  3. The findings have a limited scope. Human evaluation focuses on writing preferences, and training primarily uses small Qwen assistants and short conversations. Broader tasks, stronger models, and repeated training runs would strengthen the conclusions.

Questions for authors
  1. Can the authors include STATIC and the persona-conditioned variant in the human study, ideally under matched training budgets?

  2. Would prompt-based simulators remain weaker if they could use the same human data through retrieved examples or data-derived personas?

  3. Do the gains extend to complete task outcomes, such as completion time and editing effort, and remain stable across models and training seeds?

4 comments

We sincerely thank the reviewer for the careful reading, and for recognizing the value of our research question. We’d like to take the opportunity to discuss the weakness points and questions:


W1 / Q1 – Human study

We conducted a new human study with stronger role-playing simulators.

Setup

We tested two new pairs, each designed to isolate one factor:

  • SFTUser vs. RPUser2. Both simulators start from the real first turn and differ only in how later user turns are generated: fine-tuned on human utterances vs. role-played. This comparison directly separates the effect of the learned simulator from the effect of first-turn grounding.
  • SFTUser vs. RPUser3. This compares SFTUser with a role-playing simulator which adds persona conditioning to RPUser2.

The protocol is identical to Section 5:

  • about 90 new Prolific participants per pair, each having a conversation of at least 5 turns, resulting in ~ 450 pairwise decisions per assistant pair
  • anonymized assistant responses in randomized positions
  • 95% bootstrapped confidence intervals at the conversation level
  • participants from the original study were excluded using the feature in Prolific

Results

Table. Tie-accounted win rate; HS stands for Human Study.

ComparisonHS - Overall win rateHS - Turn1HS - Turn2HS - Turn3HS - Turn4HS - Turn5WildBench (Table 1)
SFTUser vs. Initial (original)58.1 ± 3.850.262.059.455.166.368.4 ± 2.6
RPUser1 vs. Initial (original)50.6 ± 4.245.941.054.043.756.255.4 ± 2.7
SFTUser vs. RPUser1 (original)57.3 ± 3.852.860.452.868.162.063.9 ± 2.7
SFTUser vs. RPUser2 (new) (n=89)55.9 ± 4.352.551.253.861.956.360.0 ± 2.7
SFTUser vs. RPUser3 (new) (n=93)54.0 ± 3.748.954.456.557.158.258.5 ± 2.8

Human participants prefer the SFTUser-trained assistant over both stronger role-playing variants. Against RPUser2, which shares SFTUser's real first turn, the advantage remains (55.9% win rate). So the utility of the SFTUser cannot be explained by first-turn grounding alone; it comes from how the simulator generates the user's later turns. Against RPUser3, the advantage also remains (54.0%). As on WildBench, the margin shrinks as the role-playing simulator becomes more realistic (RPUser1 → RPUser2 → RPUser3), but persona conditioning does not close the gap.

On the STATIC baseline

Thank you for this suggestion. We agree this is an important reference point, since it avoids user simulation entirely. We ran a new training expierment: single-turn RL on real WildChat conversation histories. For each prefix ot={u1,a1,…,ut}o_t = \{u_1, a_1, \dots, u_t\} of a human–LLM conversation, the assistant generates ata_t. The response is scored with the same LLM judges, rubric, and GRPO setup as our multi-turn runs, starting from Qwen2.5-3B-Instruct.

  • WildBench. The single-turn assistant beats the initial assistant (67.9 ± 2.6%) and all role-playing variants (57.3 - 63.0%). It is comparable to the SFTUser-trained assistant (49.3 ± 2.8%). WildBench scores a single response given a fixed real human context, which is exactly the single-turn training format, so this benchmark structurally favors the single-turn baseline.

  • Multi-turn interaction. In cross-simulator evaluation, the SFTUser-trained assistant wins the single-turn assistant with every test-time simulator (+0.9 to +2.2 points). In contrast, the RPUser3-trained assistant wins the single-turn baseline only when paired with role-playing simulators, and it falls behind on the held-out simulators.

Together, these results suggest that multi-turn training by itself does not win over a single-turn RL on real data. Only multi-turn training against a human-grounded simulator matches it on single-response benchmarks and exceeds it in multi-turn interaction. We will add this baseline and discussion to the revision, and we agree that including it in the human study would further strengthen the comparison.

W2 / Q2 – Additional persona baseline

We sincerely thank the reviewer for this suggestion.

On the role of the persona-conditioned role-playing simulator. We would like to clarify the scope of our claims. Our main contribution is a way of measuring simulator quality: train an assistant against the simulator, then evaluate it with real users and on real-world tasks. The simulators we study are points along a range of simulator designs chosen to show what our measurement reveals. In particular, RPUser3 is one example of a widely used design (persona-conditioned role-play), not the best possible persona-conditioned simulator. We will make this scope explicit, and appreciate the reviewer for pointing this out.

That said, we agree that a persona-conditioned simulator grounded in real data is an informative point on this range. We added RPUser4 that builds personas from the real conversation data.

Setup. For each WildChat conversation, an LLM (Qwen3-32B) reads human turns and writes a behavior profile covering: wordiness and typical message length; formality, capitalization and typos; tone; follow-up message style and conversation flow. The simulator is conditioned on this data-grounded persona. Here is an example:

- Writing Style: You write in long, detailed paragraphs, typically containing 300+ words. Your tone is formal and academic, with precise grammar and proper capitalization. You use complex sentence structures and avoid colloquial language.
- Making Requests: You provide full context and detailed instructions upfront, pasting blocks of text to be translated or paraphrased, followed by a short directive. You specify the task clearly and comprehensively in a single message.
- Reacting to the AI: You respond with a new request that is similar in form and detail to your initial one, without acknowledging or commenting on the assistant's previous response. You do not provide feedback or ask follow-up questions.
- Knowledge Background: You display intermediate to expert-level knowledge, evidenced by your ability to describe complex academic concepts and maintain consistent terminology across requests.
- Ending the Conversation: You end the conversation after two iterations of request and response.

We train a Qwen2.5-3B-Instruct assistant with RPUser4 and evaluate with WildBench-v2 and a cross-simulator evaluation.

Results (WildBench-v2)

Assistant trained withChecklist satisfaction [%]Win rate vs. RPUser3 [%]Win rate vs. RPUser4 [%]Win rate vs. SFTUser [%]
RPUser357.7 ± 1.0—47.3 ± 2.641.5 ± 2.8
RPUser457.9 ± 0.952.7 ± 2.6—43.9 ± 2.7
SFTUser61.5 ± 0.958.5 ± 2.856.1 ± 2.7—

Results (cross-simulation evaluation)

Eval with ↓ Train with →RPUser3RPUser4SFTUser
RPUser383.8 ± 0.281.9 ± 0.384.8 ± 0.2
RPUser484.5 ± 0.284.9 ± 0.284.8 ± 0.2
SFTUser80.3 ± 0.380.2 ± 0.382.5 ± 0.3
SFTUser-WC-4.8M76.8 ± 0.380.3 ± 0.382.8 ± 0.2
SFTUser-LMSYS68.3 ± 0.468.7 ± 0.470.2 ± 0.4

Grounding personas in real data closes almost none of the gap in WildBench. The RPUser4-trained assistant improves over RPUser3 (52.7% win rate) and recovers 0.2 percentage point of the RPUser3-to-SFTUser gap in checklist satisfaction. However, it still trails SFTUser (43.9% win rate vs. SFTUser). With the same human data and a profile drawn from each user’s own conversation, prompting closes almost none of the gap compared to fine-tuning.

W3 / Q3 – Variance across RL training runs, and other models

To measure how much training seed variance affects our results, we trained two more RL runs each for RPUser3 and SFTUser, giving three independent runs per simulator including the original run. We chose RPUser3 because it is the strongest of the original role-playing variants and has the smallest gap to SFTUser (58.5% WildBench win rate). Random seed affects rollout sampling and the order of training data.

(a) Between-simulator comparison over all seed pairs.

SFTUser's WildBench win rate against RPUser3, for every pair of runs. Run 1 is the original run in the paper.

RPUser3 ↓ / SFTUser →Run 1Run 2Run 3
Run 158.5 ± 2.858.2 ± 2.758.0 ± 2.6
Run 256.6 ± 2.655.9 ± 2.756.6 ± 2.7
Run 357.8 ± 2.757.2 ± 2.758.1 ± 2.7

(b) Stability of the metrics. Values for each of the three runs.

Train SimulatorWB satisfaction [%]--Cross-sim eval (with RPUser3)--
-Run 1Run 2Run 3Run 1Run 2Run 3
RPUser357.7 ± 1.058.5 ± 0.958.3 ± 1.083.8 ± 0.284.0 ± 0.283.7 ± 0.3
SFTUser61.5 ± 0.961.3 ± 0.861.5 ± 0.984.8 ± 0.284.8 ± 0.284.4 ± 0.2

Result. Runs of the same simulator are nearly indistinguishable.

(a) The difference between simulators holds across seeds: SFTUser-trained assistant outperforms RPUser3-trained assistant in all 9 seed pairs, with win rates from 55.9 ± 2.7 to 58.5 ± 2.8%. The original 58.5 ± 2.8% therefore reflects a valid range of performance. We will report the full table in the revised version.

(b) Both metrics, the WildBench checklist satisfaction rate and the cross-simulator evaluation is also stable across seeds. SFTUser’s satisfaction ranges from 61.3 ± 0.8 to 61.5 ± 0.9% and RPUser3’s from 57.7 ± 1.0 to 58.5 ± 0.9%, so the two ranges do not overlap. Cross-simulator evaluation shows the same separation.

Also, we would like to note that training results are available for different model family (Table 9) and larger assistants (Table 8).

Thank you for the follow-up and the additional experiments. I have no further questions.

Reviewer_BdccOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to evaluate user simulators based on their downstream utility, namely how well assistants trained with these simulators perform with real users. It compares prompted and fine-tuned simulators under a shared RL setup, using automated benchmarks, cross-simulator evaluation, and a human study with 283 participants. The results suggest that simulators trained on human conversations are more effective, and that stronger role-playing ability does not necessarily translate into better training environments.

Critical strengths
  1. The paper addresses an interesting and important problem: how should we evaluate user simulators for training collaborative assistants? Evaluating the resulting assistants with real users is a reasonable and meaningful approach.

  2. The experiments are fairly comprehensive, combining human evaluation with automated benchmarks across different model sizes and families. The human study is also carefully designed, with anonymous comparisons, randomized response positions, and confidence intervals.

  3. The paper provides several interesting findings beyond the main results, particularly regarding how prompted and fine-tuned simulators scale differently and how performance on the training simulator may not transfer to other users.

Critical weaknesses
  1. The human evaluation is somewhat limited. It only covers writing tasks and compares against RPUser1, leaving the comparison with the stronger RPUser3 unverified. Also, since both assistants continue from the selected response, the evaluation focuses on turn-level preferences rather than the quality of complete conversations.

  2. The evaluation cost is relatively high. Each simulator requires training a separate assistant, which makes this approach expensive to scale. While Appendix I.1 explores some fidelity metrics, it remains unclear whether these could serve as reliable proxies for downstream performance, especially for unseen simulators.

  3. The results may be affected by RL training variance. The paper does not report repeated training runs, so the confidence intervals do not account for this source of randomness. Repeating some key experiments would help establish how robust the findings are, especially for comparisons with smaller performance gaps.

Questions for authors

How do you verify that the inferred user intents are accurate?

3 comments

We sincerely thank the reviewer for the careful reading, and for recognizing the value of our research question. We’d like to take the opportunity to discuss the weakness points and questions:


Weakness 1 – Human study

We conducted a new human study with stronger role-playing simulators.

Setup

We tested two new pairs, each designed to isolate one factor:

  • SFTUser vs. RPUser2. Both simulators start from the real first turn and differ only in how later user turns are generated: fine-tuned on human utterances vs. role-played. This comparison directly separates the effect of the learned simulator from the effect of first-turn grounding.
  • SFTUser vs. RPUser3. This compares SFTUser with a role-playing simulator which adds persona conditioning to RPUser2.

The protocol is identical to Section 5:

  • about 90 new Prolific participants per pair, each having a conversation of at least 5 turns
  • anonymized assistant responses in randomized positions
  • 95% bootstrapped confidence intervals at the conversation level
  • participants from the original study were excluded using the feature in Prolific

Results

Table. Tie-accounted win rate; HS stands for Human Study.

ComparisonHS - Overall win rateHS - Turn1HS - Turn2HS - Turn3HS - Turn4HS - Turn5WildBench (Table 1)
SFTUser vs. Initial (original)58.1 ± 3.850.262.059.455.166.368.4 ± 2.6
RPUser1 vs. Initial (original)50.6 ± 4.245.941.054.043.756.255.4 ± 2.7
SFTUser vs. RPUser1 (original)57.3 ± 3.852.860.452.868.162.063.9 ± 2.7
SFTUser vs. RPUser2 (new) (n=89)55.9 ± 4.352.551.253.861.956.360.0 ± 2.7
SFTUser vs. RPUser3 (new) (n=93)54.0 ± 3.748.954.456.557.158.258.5 ± 2.8

Human participants prefer the SFTUser-trained assistant over both stronger role-playing variants. Against RPUser2, which shares SFTUser's real first turn, the advantage remains (55.9% win rate). So the utility of the SFTUser cannot be explained by first-turn grounding alone; it comes from how the simulator generates the user's later turns. Against RPUser3, the advantage also remains (54.0%). As on WildBench, the margin shrinks as the role-playing simulator becomes more realistic (RPUser1 → RPUser2 → RPUser3), but persona conditioning does not close the gap.

On the shared-history design

We chose the turn-level pairwise decision design so that both assistants are always compared in the same state. This allows a matched comparison at every turn and gives enough statistical power within the human study budget. Independent conversations would differ in conversation topic, path and length, which makes comparisons much noisier. We agree the design measures preference turn by turn, not the outcome of a whole collaboration. Two points might help here:

  • Results broken down by turn (Figure 3, right and our new table with additional assistant pairs) show the preference persists into later turns, showing that preference toward SFTUser-trained assistant emerges and persists in later turns.
  • Cross-simulator evaluation (Table 3) score assistants in multi-turn settings.

We will state this limitation explicitly. A study measuring outcomes of whole independent conversations, such as satisfaction with the final draft and number of turns until satisfied, is a valuable direction for future work.


Weakness 2 – The evaluation cost is relatively high

We agree that training an assistant for each user simulator is costly. We see this cost as necessary: we define a simulator’s utility as the quality of the assistant trained with it, and Appendix I.1 shows that cheaper fidelity proxies do not reliably track it. Reward models in RLHF are in a similar position: their accuracy on held-out preference data correlates only weakly with the quality of the policy trained on them, so they are ultimately judged by the trained policy [1]. Developing a proxy metric that can predict utility for unseen simulators remains an important research question. Our framework provides the downstream ground truth needed to test such proxies, and we will expand the future-work discussion of proxies (Section 7) in the revision.

[1] Rethinking Reward Model Evaluation: Are We Barking Up the Wrong Tree?, Wen et. al., ICLR 2025.

Weakness 3 – Multiple training seeds

To measure how much training seed variance affects our results, we trained two more RL runs each for RPUser3 and SFTUser, giving three independent runs per simulator including the original run. We chose RPUser3 because it is the strongest of the original role-playing variants and has the smallest gap to SFTUser (58.5% WildBench win rate). Random seed affects rollout sampling and the order of training data.

(a) Between-simulator comparison over all seed pairs.

SFTUser's WildBench win rate against RPUser3, for every pair of runs. Run 1 is the original run in the paper.

RPUser3 ↓ / SFTUser →Run 1Run 2Run 3
Run 158.5 ± 2.858.2 ± 2.758.0 ± 2.6
Run 256.6 ± 2.655.9 ± 2.756.6 ± 2.7
Run 357.8 ± 2.757.2 ± 2.758.1 ± 2.7

(b) Stability of the metrics. Values for each of the three runs.

Train SimulatorWB satisfaction [%]--Cross-sim eval (with RPUser3)--
-Run 1Run 2Run 3Run 1Run 2Run 3
RPUser357.7 ± 1.058.5 ± 0.958.3 ± 1.083.8 ± 0.284.0 ± 0.283.7 ± 0.3
SFTUser61.5 ± 0.961.3 ± 0.861.5 ± 0.984.8 ± 0.284.8 ± 0.284.4 ± 0.2

Result. Runs of the same simulator are nearly indistinguishable.

(a) The difference between simulators holds across seeds: SFTUser-trained assistant outperforms RPUser3-trained assistant in all 9 seed pairs, with win rates from 55.9 ± 2.7 to 58.5 ± 2.8%. The original 58.5 ± 2.8% therefore reflects a valid range of performance. We will report the full table in the revised version.

(b) Both metrics, the WildBench checklist satisfaction rate and the cross-simulator evaluation is also stable across seeds. SFTUser’s satisfaction ranges from 61.3 ± 0.8 to 61.5 ± 0.9% and RPUser3’s from 57.7 ± 1.0 to 58.5 ± 0.9%, so the two ranges do not overlap. Cross-simulator evaluation shows the same separation.


Question 1 – Accuracy of the inferred user intents

We checked whether the generated intents are accurate and discuss its impacts on our conclusion.

Setup. We sampled 500 conversations randomly from the training split. Claude Sonnet 5.5, a model different from Qwen3-32B that generated the intents, compared each intent against the real conversation and assigned one of three labels:

  • Faithful: the intent states the user's main goal correctly.
  • Partially faithful: the main goal is right, but the intent misses a secondary goal, or the user's goal changed partway through and the intent does not reflect that.
  • Unfaithful: the intent states the wrong goal.

Results ([,] indicates a 95% confidence interval):

All (n=500)
Faithful88.2% [85.1, 90.7]
Partially faithful10.2% [7.8, 13.2]
Unfaithful1.6% [0.8, 3.1]

Checking the judge. A lenient judge that accepts any plausible intent would also report a high faithful rate. To rule this out, we ran a negative control: the same 500 conversations, each paired with the intent of a randomly chosen conversation, with the same judge and instruction. The judge labeled 498 of the 500 mismatched intents unfaithful (99.6%, 95% CI [98.6, 99.9]). In the two exceptions, we observed that the borrowed intent came from a conversation with nearly the same goal. The high faithful rate above therefore reflects intents that match their conversations, not a lenient judge.

Additionally, author manually annotated 50 downsampled intents. Disagreements with Sonnet occurred in two cases: in one case, Sonnet annotated faithful but author annotated partially faithful, and vice versa in another case, giving Cohen's Kappa of 0.81.

Effect on our conclusions. Every user simulator is conditioned on the same intents, so intent errors do not favor any simulator in our comparisons. Moreover, the human study and WildBench do not use the intents at all: participants set their own writing goals, and WildBench supplies its own tasks.

Dear Reviewer,

Thank you again for your time and feedback on our paper. Since the discussion period ends tonight, we wanted to check whether our responses have addressed your concerns. If you have any remaining questions, we would be happy to respond. If our responses have resolved your concerns, we would be grateful if you could consider updating your assessment.

Best regards, The authors

Reviewer_5ftJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether the quality of a user simulator can be measured through its downstream utility, that is, how well an assistant trained against it performs with real users. The authors compare role-playing LLM simulators with a simulator fine-tuned on human utterances from WildChat, using multi-turn GRPO (with LLM-judged rewards) to train assistants. They evaluate the trained assistants on WildBench, cross-simulator generalization, and a human study with 283 participants. The results show that assistants trained with the fine-tuned simulator consistently outperform those trained with role-playing (prompted) simulators on automated evaluations. The paper also studies how simulator design and model size affect downstream performance.

Critical strengths
  • Well-motivated and practically relevant. The paper addresses an important question: whether user simulators actually help train assistants that perform better with real humans. This is particularly relevant as simulated interactions are increasingly used for training interactive agents.

  • Useful insights from empirical evaluation. The paper combines automated benchmarks, cross-simulator evaluation, and a human study with 283 participants. The results consistently favor the fine-tuned simulator, showing interesting patterns around how the choice of user simulator affects downstream assistant performance.

Critical weaknesses
  • The paper defines simulator quality through the performance of assistants interacting with real humans, but the human study only compares SFT USER against the simplest role-playing baseline (RP USER1), leaving out stronger variants (RP USER2 and RP USER3) that perform substantially better on WildBench.

    • On WildBench (Table 1), simply using real opening prompts (RPuser2) and adding personas (RPuser3) closes most of the performance gap between RPuser1 and SFTuser. By only testing RPuser1 in the human study, the authors mix up two different things: the benefit of fine-tuning a user simulator versus the benefit of starting RL training with real human prompts. It is entirely unclear if SFTuser would still beat RPuser2 or RPuser3 when tested with real humans.

    • The user study design seems a bit confusing. The humans are made to choose a response at each turn and continue along a shared conversation history, rather than interacting independently with each assistant. This establishes turn-level human preference but provides weaker evidence for improvements in end-to-end collaboration. Evaluating stronger baselines through independent multi-turn conversations would substantially strengthen the central claim.

  • Simulator comparisons:

    • Differences in rollout length and termination behavior: the authors show that role-playing simulators almost never output the <|endconversation|> stop token, whereas SFTuser frequently ends conversations early. Because of this, RPuser3 training rollouts are more than twice as long as SFTuser rollouts (Figures 5,6). Because the RL algorithm only gives a single score at the very end of the chat, RPuser forces the assistant to keep chatting even after the user's goal is met, whereas SFTuser gives the assistant shorter, much easier episodes to learn from. The reported experiments do not isolate this effect. Matching rollout lengths or termination behavior across simulators would help determine whether the performance gap persists.

    • While the paper claims to evaluate persona conditioning by citing large-scale persona literature, the Appendix shows that RPuser3 only samples uniformly from three hardcoded behavioral prompts (plus the default): Tangential, Impatient, and Incomplete Utterances. Rather than capturing realistic human communication styles or data-driven user profiles from WildChat, these prompts seem to enforce some self-curated rules. Concluding that persona conditioning cannot close the gap to SFTuser based on just three handcrafted, uniformly sampled prompt templates might need more rigorous/realistic evidence. A stronger data-grounded persona baseline would make this comparison more convincing.

Questions for authors
  1. How much of the downstream performance gap can be attributed to differences in conversation lengths?

  2. How stable are the downstream results across independent RL training runs? In particular, are the differences between SFT USER and the stronger role-playing variants consistent across training seeds?

  3. How does the RL judge prompt (Appendix E) score multi-turn rollouts when its prompt asks to evaluate a single response?

  4. As the assistant policy changes during training, does the learned simulator remain realistic when responding to increasingly unfamiliar assistant behaviors? Could distribution shift within the training process affect the conclusions?

8 comments

We sincerely thank the reviewer for the careful reading, and for recognizing the value of our research question. We’d like to take the opportunity to discuss the weakness points and questions:


Weakness - human study only compares SFTUser against the simplest role-playing

We conducted a new human study with stronger role-playing simulators.

Setup

We tested two new pairs, each designed to isolate one factor:

  • SFTUser vs. RPUser2. Both simulators start from the real first turn and differ only in how later user turns are generated: fine-tuned on human utterances vs. role-played. This comparison directly separates the effect of the learned simulator from the effect of first-turn grounding.
  • SFTUser vs. RPUser3. This compares SFTUser with a role-playing simulator which adds persona conditioning to RPUser2.

The protocol is identical to Section 5:

  • about 90 new Prolific participants per pair, each having a conversation of at least 5 turns, resulting in ~ 450 pairwise decisions per assistant pair
  • anonymized assistant responses in randomized positions
  • 95% bootstrapped confidence intervals at the conversation level
  • participants from the original study were excluded using the feature in Prolific

Results

Table. Tie-accounted win rate; HS stands for Human Study.

ComparisonHS - Overall win rateHS - Turn1HS - Turn2HS - Turn3HS - Turn4HS - Turn5WildBench (Table 1)
SFTUser vs. Initial (original)58.1 ± 3.850.262.059.455.166.368.4 ± 2.6
RPUser1 vs. Initial (original)50.6 ± 4.245.941.054.043.756.255.4 ± 2.7
SFTUser vs. RPUser1 (original)57.3 ± 3.852.860.452.868.162.063.9 ± 2.7
SFTUser vs. RPUser2 (new) (n=89)55.9 ± 4.352.551.253.861.956.360.0 ± 2.7
SFTUser vs. RPUser3 (new) (n=93)54.0 ± 3.748.954.456.557.158.258.5 ± 2.8

Human participants prefer the SFTUser-trained assistant over both stronger role-playing variants. Against RPUser2, which shares SFTUser's real first turn, the advantage remains (55.9% win rate). So the utility of the SFTUser cannot be explained by first-turn grounding alone; it comes from how the simulator generates the user's later turns. Against RPUser3, the advantage also remains (54.0%). As on WildBench, the margin shrinks as the role-playing simulator becomes more realistic (RPUser1 → RPUser2 → RPUser3), but persona conditioning does not close the gap.

On the shared-history design

We chose the turn-level pairwise decision design so that both assistants are always compared in the same state. This allows a matched comparison at every turn and gives enough statistical power within the human study budget. Independent conversations would differ in conversation topic, path and length, which makes comparisons much noisier. We agree the design measures preference turn by turn, not the outcome of a whole collaboration. Two points might help here:

  • Results broken down by turn (Figure 3, right and our new table with additional assistant pairs) show the preference persists into later turns, showing that preference toward SFTUser-trained assistant emerges and persists in later turns.
  • Cross-simulator evaluation (Table 3) score assistants in multi-turn settings.

We will state this limitation explicitly. A study measuring outcomes of whole independent conversations, such as satisfaction with the final draft and number of turns until satisfied, is a valuable direction for future work.

Weakness - Unmatched rollout lengths + Question 1

We ran an experiment to isolate the effect of prolonged conversation length in RL runs with role-playing simulators.

Setup. We matched the conversation length distribution of the role-playing simulators to that of real users. First, we computed the distribution of conversation turn counts in WildChat-1M, restricted to our 5-turn maximum: 1 turn 59.3%, 2 turns 20.3%, 3 turns 10.4%, 4 turns 6.1%, 5 turns 3.9%. During each RL rollout, we sample a target length from this distribution and stop the user simulator once that length is reached. Everything else is identical to the original runs. We retrained the assistant with RPUser2 and with RPUser3 under this length matching (RPUser2-LM and RPUser3-LM).

Training simulatorAvg. turns / rolloutChecklist satisfaction [%]win rate vs. SFTUser [%]
RPUser24.9956.6 ± 1.040.0 ± 2.7
RPUser2-LM1.7456.3 ± 1.039.3 ± 2.6
RPUser34.9957.7 ± 1.041.5 ± 2.8
RPUser3-LM1.7457.2 ± 0.940.8 ± 2.7
SFTUser1.9261.5 ± 0.9—

Result. Length matching has minimal impact on the performance of the trained assistant. RPUser2-LM and RPUser3-LM score comparably to their unmatched counterparts (checklist satisfaction 56.3 vs. 56.6 and 57.2 vs. 57.7), and both remain below SFTUser (61.5). SFTUser still wins 60.7% and 59.2% in head-to-head comparisons. Most of the advantage therefore comes from what the simulator says (so that how it forms the conversation trajectory), not from how long the conversations last.

We note that this setup matches conversation length, but does not make the role-playing simulators end conversations at the right moment, such as when the user's goal is met. We see this remaining difference as part of what our utility-based evaluation measures, not as a confound. Ending a conversation when one's need is satisfied is a real feature of human users. A simulator that captures it gives the assistant a more faithful training signal. What this experiment rules out is the simpler explanation that SFTUser helps only because it produces shorter, easier episodes.

Weakness - Additional persona baseline

We sincerely thank the reviewer for this suggestion.

On the role of the persona-conditioned role-playing simulator. We would like to clarify the scope of our claims. Our main contribution is a way of measuring simulator quality: train an assistant against the simulator, then evaluate it with real users and on real-world tasks. The simulators we study are points along a range of simulator designs chosen to show what our measurement reveals. In particular, RPUser3 is one example of a widely used design (persona-conditioned role-play), not the best possible persona-conditioned simulator. We will make this scope explicit, and appreciate the reviewer for pointing this out.

That said, we agree that a persona-conditioned simulator grounded in real data is an informative point on this range. We added RPUser4 that builds personas from the real conversation data.

Setup. For each WildChat conversation, an LLM (Qwen3-32B) reads human turns and writes a behavior profile covering: wordiness and typical message length; formality, capitalization and typos; tone; follow-up message style and conversation flow. The simulator is conditioned on this data-grounded persona. Here is an example:

- Writing Style: You write in long, detailed paragraphs, typically containing 300+ words. Your tone is formal and academic, with precise grammar and proper capitalization. You use complex sentence structures and avoid colloquial language.
- Making Requests: You provide full context and detailed instructions upfront, pasting blocks of text to be translated or paraphrased, followed by a short directive. You specify the task clearly and comprehensively in a single message.
- Reacting to the AI: You respond with a new request that is similar in form and detail to your initial one, without acknowledging or commenting on the assistant's previous response. You do not provide feedback or ask follow-up questions.
- Knowledge Background: You display intermediate to expert-level knowledge, evidenced by your ability to describe complex academic concepts and maintain consistent terminology across requests.
- Ending the Conversation: You end the conversation after two iterations of request and response.

We train a Qwen2.5-3B-Instruct assistant with RPUser4 and evaluate with WildBench-v2 and a cross-simulator evaluation.

Results (WildBench-v2)

Assistant trained withChecklist satisfaction [%]Win rate vs. RPUser3 [%]Win rate vs. RPUser4 [%]Win rate vs. SFTUser [%]
RPUser357.7 ± 1.0—47.3 ± 2.641.5 ± 2.8
RPUser457.9 ± 0.952.7 ± 2.6—43.9 ± 2.7
SFTUser61.5 ± 0.958.5 ± 2.856.1 ± 2.7—

Results (cross-simulation evaluation)

Eval with ↓ Train with →RPUser3RPUser4SFTUser
RPUser383.8 ± 0.281.9 ± 0.384.8 ± 0.2
RPUser484.5 ± 0.284.9 ± 0.284.8 ± 0.2
SFTUser80.3 ± 0.380.2 ± 0.382.5 ± 0.3
SFTUser-WC-4.8M76.8 ± 0.380.3 ± 0.382.8 ± 0.2
SFTUser-LMSYS68.3 ± 0.468.7 ± 0.470.2 ± 0.4

Grounding personas in real data closes almost none of the gap in WildBench. The RPUser4-trained assistant improves over RPUser3 (52.7% win rate) and recovers 0.2 percentage point of the RPUser3-to-SFTUser gap in checklist satisfaction. However, it still trails SFTUser (43.9% win rate vs. SFTUser). With the same human data and a profile drawn from each user’s own conversation, prompting closes almost none of the gap compared to fine-tuning.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Discussion (as of November 18 19:00 AoE time)

We thank the reviewers for their constructive feedback. Reviewers agreed that measuring user simulators by their downstream utility with real users is an important question, and that the controlled experimental design is sound. The reviewers suggested applying our measurement to more simulator designs and further testing its robustness. In response, we ran the following additional experiments:

  • More simulator designs in the human study (5ftJ, Bdcc, C9Nw): We extended the user study to cover every simulator design in our comparison. Downstream utility with real users increases as the simulator becomes more grounded in user data, and the human study and the benchmark agree on this ordering.
  • Data-grounded personas (5ftJ, C9Nw): We added a role-playing simulator conditioned on each user's real conversational style. It adds another point to the range of simulators we study, and broaden the empricial results of our measurement protocol.
  • Robustness of the measurement (5ftJ, Bdcc, C9Nw): The results are stable across independent training runs (consistent in all 9 seed pairs), and empricial results are unchanged when conversation lengths are controlled.

We also answered clarifying questions on the study design, the judge prompt, evaluation cost, and the accuracy of inferred intents. The revision will add the new results and adopt clearer simulator names as suggested by Reviewer 5ftJ.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.