- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper asks a practical question: does training with simulated users actually make assistants better for real people? It compares role-playing, persona-conditioned, and human-data-fine-tuned simulators through controlled training experiments, automated evaluations, and a study with 283 participants. The main finding is that improvements during simulated training do not necessarily transfer to human interactions, while training with a fine-tuned user simulator produces measurable gains in human preference.
Critical strengths
-
The paper evaluates what ultimately matters. It measures simulator quality through its impact on trained assistants and real users, rather than relying solely on how realistic simulated conversations appear.
-
The comparisons are carefully controlled. Keeping the assistant, training algorithm, and reward setup consistent makes the effects of simulator choice easier to assess. Cross-simulator testing adds useful evidence about transfer.
-
The human study provides meaningful validation. The fine-tuned-simulator-trained assistant achieves a 58.1% preference win rate against the initial assistant, while ordinary role-playing training shows no significant advantage. Reporting both positive and negative findings makes the study informative for practitioners.
Critical weaknesses
-
The human study misses two important baselines. Neither the persona-conditioned variant nor STATIC is included. STATIC trains directly on real conversation histories and performs similarly to the fine-tuned simulator approach on WildBench, leaving the additional value of multi-turn simulation insufficiently established.
-
The persona baseline is limited. A small set of behavioral prompts does not represent the full potential of persona-based simulation. Since the fine-tuned simulator also receives substantial human supervision, the results do not fully separate the benefits of fine-tuning from access to human behavioral data.
-
The findings have a limited scope. Human evaluation focuses on writing preferences, and training primarily uses small Qwen assistants and short conversations. Broader tasks, stronger models, and repeated training runs would strengthen the conclusions.
Questions for authors
-
Can the authors include STATIC and the persona-conditioned variant in the human study, ideally under matched training budgets?
-
Would prompt-based simulators remain weaker if they could use the same human data through retrieved examples or data-derived personas?
-
Do the gains extend to complete task outcomes, such as completion time and editing effort, and remain stable across models and training seeds?
4 comments
W2 / Q2 – Additional persona baseline
We sincerely thank the reviewer for this suggestion.
On the role of the persona-conditioned role-playing simulator. We would like to clarify the scope of our claims. Our main contribution is a way of measuring simulator quality: train an assistant against the simulator, then evaluate it with real users and on real-world tasks. The simulators we study are points along a range of simulator designs chosen to show what our measurement reveals. In particular, RPUser3 is one example of a widely used design (persona-conditioned role-play), not the best possible persona-conditioned simulator. We will make this scope explicit, and appreciate the reviewer for pointing this out.
That said, we agree that a persona-conditioned simulator grounded in real data is an informative point on this range. We added RPUser4 that builds personas from the real conversation data.
Setup. For each WildChat conversation, an LLM (Qwen3-32B) reads human turns and writes a behavior profile covering: wordiness and typical message length; formality, capitalization and typos; tone; follow-up message style and conversation flow. The simulator is conditioned on this data-grounded persona. Here is an example:
- Writing Style: You write in long, detailed paragraphs, typically containing 300+ words. Your tone is formal and academic, with precise grammar and proper capitalization. You use complex sentence structures and avoid colloquial language.
- Making Requests: You provide full context and detailed instructions upfront, pasting blocks of text to be translated or paraphrased, followed by a short directive. You specify the task clearly and comprehensively in a single message.
- Reacting to the AI: You respond with a new request that is similar in form and detail to your initial one, without acknowledging or commenting on the assistant's previous response. You do not provide feedback or ask follow-up questions.
- Knowledge Background: You display intermediate to expert-level knowledge, evidenced by your ability to describe complex academic concepts and maintain consistent terminology across requests.
- Ending the Conversation: You end the conversation after two iterations of request and response.
We train a Qwen2.5-3B-Instruct assistant with RPUser4 and evaluate with WildBench-v2 and a cross-simulator evaluation.
Results (WildBench-v2)
| Assistant trained with | Checklist satisfaction [%] | Win rate vs. RPUser3 [%] | Win rate vs. RPUser4 [%] | Win rate vs. SFTUser [%] |
|---|---|---|---|---|
| RPUser3 | 57.7 ± 1.0 | — | 47.3 ± 2.6 | 41.5 ± 2.8 |
| RPUser4 | 57.9 ± 0.9 | 52.7 ± 2.6 | — | 43.9 ± 2.7 |
| SFTUser | 61.5 ± 0.9 | 58.5 ± 2.8 | 56.1 ± 2.7 | — |
Results (cross-simulation evaluation)
| Eval with ↓ Train with → | RPUser3 | RPUser4 | SFTUser |
|---|---|---|---|
| RPUser3 | 83.8 ± 0.2 | 81.9 ± 0.3 | 84.8 ± 0.2 |
| RPUser4 | 84.5 ± 0.2 | 84.9 ± 0.2 | 84.8 ± 0.2 |
| SFTUser | 80.3 ± 0.3 | 80.2 ± 0.3 | 82.5 ± 0.3 |
| SFTUser-WC-4.8M | 76.8 ± 0.3 | 80.3 ± 0.3 | 82.8 ± 0.2 |
| SFTUser-LMSYS | 68.3 ± 0.4 | 68.7 ± 0.4 | 70.2 ± 0.4 |
Grounding personas in real data closes almost none of the gap in WildBench. The RPUser4-trained assistant improves over RPUser3 (52.7% win rate) and recovers 0.2 percentage point of the RPUser3-to-SFTUser gap in checklist satisfaction. However, it still trails SFTUser (43.9% win rate vs. SFTUser). With the same human data and a profile drawn from each user’s own conversation, prompting closes almost none of the gap compared to fine-tuning.
W3 / Q3 – Variance across RL training runs, and other models
To measure how much training seed variance affects our results, we trained two more RL runs each for RPUser3 and SFTUser, giving three independent runs per simulator including the original run. We chose RPUser3 because it is the strongest of the original role-playing variants and has the smallest gap to SFTUser (58.5% WildBench win rate). Random seed affects rollout sampling and the order of training data.
(a) Between-simulator comparison over all seed pairs.
SFTUser's WildBench win rate against RPUser3, for every pair of runs. Run 1 is the original run in the paper.
| RPUser3 ↓ / SFTUser → | Run 1 | Run 2 | Run 3 |
|---|---|---|---|
| Run 1 | 58.5 ± 2.8 | 58.2 ± 2.7 | 58.0 ± 2.6 |
| Run 2 | 56.6 ± 2.6 | 55.9 ± 2.7 | 56.6 ± 2.7 |
| Run 3 | 57.8 ± 2.7 | 57.2 ± 2.7 | 58.1 ± 2.7 |
(b) Stability of the metrics. Values for each of the three runs.
| Train Simulator | WB satisfaction [%] | - | - | Cross-sim eval (with RPUser3) | - | - |
|---|---|---|---|---|---|---|
| - | Run 1 | Run 2 | Run 3 | Run 1 | Run 2 | Run 3 |
| RPUser3 | 57.7 ± 1.0 | 58.5 ± 0.9 | 58.3 ± 1.0 | 83.8 ± 0.2 | 84.0 ± 0.2 | 83.7 ± 0.3 |
| SFTUser | 61.5 ± 0.9 | 61.3 ± 0.8 | 61.5 ± 0.9 | 84.8 ± 0.2 | 84.8 ± 0.2 | 84.4 ± 0.2 |
Result. Runs of the same simulator are nearly indistinguishable.
(a) The difference between simulators holds across seeds: SFTUser-trained assistant outperforms RPUser3-trained assistant in all 9 seed pairs, with win rates from 55.9 ± 2.7 to 58.5 ± 2.8%. The original 58.5 ± 2.8% therefore reflects a valid range of performance. We will report the full table in the revised version.
(b) Both metrics, the WildBench checklist satisfaction rate and the cross-simulator evaluation is also stable across seeds. SFTUser’s satisfaction ranges from 61.3 ± 0.8 to 61.5 ± 0.9% and RPUser3’s from 57.7 ± 1.0 to 58.5 ± 0.9%, so the two ranges do not overlap. Cross-simulator evaluation shows the same separation.
Also, we would like to note that training results are available for different model family (Table 9) and larger assistants (Table 8).
Thank you for the follow-up and the additional experiments. I have no further questions.
We sincerely thank the reviewer for the careful reading, and for recognizing the value of our research question. We’d like to take the opportunity to discuss the weakness points and questions:
W1 / Q1 – Human study
We conducted a new human study with stronger role-playing simulators.
Setup
We tested two new pairs, each designed to isolate one factor:
The protocol is identical to Section 5:
Results
Table. Tie-accounted win rate; HS stands for Human Study.
Human participants prefer the SFTUser-trained assistant over both stronger role-playing variants. Against RPUser2, which shares SFTUser's real first turn, the advantage remains (55.9% win rate). So the utility of the SFTUser cannot be explained by first-turn grounding alone; it comes from how the simulator generates the user's later turns. Against RPUser3, the advantage also remains (54.0%). As on WildBench, the margin shrinks as the role-playing simulator becomes more realistic (RPUser1 → RPUser2 → RPUser3), but persona conditioning does not close the gap.
On the STATIC baseline
Thank you for this suggestion. We agree this is an important reference point, since it avoids user simulation entirely. We ran a new training expierment: single-turn RL on real WildChat conversation histories. For each prefix ot={u1,a1,…,ut} of a human–LLM conversation, the assistant generates at. The response is scored with the same LLM judges, rubric, and GRPO setup as our multi-turn runs, starting from Qwen2.5-3B-Instruct.
WildBench. The single-turn assistant beats the initial assistant (67.9 ± 2.6%) and all role-playing variants (57.3 - 63.0%). It is comparable to the SFTUser-trained assistant (49.3 ± 2.8%). WildBench scores a single response given a fixed real human context, which is exactly the single-turn training format, so this benchmark structurally favors the single-turn baseline.
Multi-turn interaction. In cross-simulator evaluation, the SFTUser-trained assistant wins the single-turn assistant with every test-time simulator (+0.9 to +2.2 points). In contrast, the RPUser3-trained assistant wins the single-turn baseline only when paired with role-playing simulators, and it falls behind on the held-out simulators.
Together, these results suggest that multi-turn training by itself does not win over a single-turn RL on real data. Only multi-turn training against a human-grounded simulator matches it on single-response benchmarks and exceeds it in multi-turn interaction. We will add this baseline and discussion to the revision, and we agree that including it in the human study would further strengthen the comparison.