- Rating
- 1
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies how learned user representations can improve the behavioral fidelity of LLM-based user simulators. It separates representations into task and behavior descriptions and introduces a conversation reconstruction framework, where simulated interactions are compared with original conversations to generate hindsight feedback. The authors use this feedback to train UserRepGen-4B through online supervised fine-tuning. Experiments across seven datasets report improvements in reconstruction fidelity, with additional analyses examining cross-task transfer, population-level behavioral diversity, and user identification across conversations.
Critical strengths
The paper addresses a meaningful problem in user simulation: improving simulation fidelity by refining user representations rather than solely optimizing the simulator itself. Separating task descriptions from behavioral descriptions provides a clear representation interface that could potentially be reused across tasks.
The paper is clearly and concisely written, making it easy to follow.
Critical weaknesses
First, the paper primarily relies on GPT-5.4 mini to assess whether simulated users resemble real humans, without sufficiently establishing its ability to judge behavioral fidelity. The evaluation emphasizes linguistic style, formatting, response length, and turn count, making it difficult to distinguish genuine behavioral improvements from adaptation to the scoring rubric. Higher scores do not necessarily imply more realistic human behavior. The reliance on a relatively weak judge is particularly concerning given the lack of evaluation using stronger models.
Second, although the authors report human evaluation, the study in the appendix mainly tests agreement between human preferences and the LLM judge’s rankings, while excluding cases where the judge assigns equal scores. This does not substitute for user studies and A/B testing with real participants, nor does it demonstrate that simulated users respond as real users would in new situations. Insufficient reporting of annotator numbers, inter-annotator agreement, and verifiable annotation records also makes the reliability of this evidence difficult to assess independently.
Finally, although the paper includes supplementary analysis using an existing distributional-gap metric, its main results still rely on an author-designed conversation reconstruction protocol. The paper lacks validation on independent evaluation resources and comparisons with existing user-profile construction, behavioral representation, and specialized simulation methods and models. Furthermore, the experiments do not sufficiently disentangle the contributions of training data, prompt design, representation length, and the proposed algorithm. Consequently, the claimed advantage over GPT-5.5 remains limited to the authors’ evaluation protocol and does not establish a genuine advantage in realistic user simulation.
Questions for authors
The authors may not have had sufficient time to complete a more thorough evaluation. I encourage them to prioritize addressing the concerns raised in the Weaknesses section. If additional experiments and explanations adequately resolve these concerns, I would be willing to raise my score.
One additional question concerns Section 7.2: with 700 samples per group and 500 clusters, the estimated JS divergence may be sensitive to finite-sample effects and clustering randomness. Could the authors report results across different random seeds and cluster counts, along with confidence intervals, to verify that the reported reduction in the distributional gap is robust?
4 comments
Rebuttal by Authors (2/n)
W2. Human Validation of LLM Judges
We agree with the reviewer that our human validation study is not a substitute for user studies or A/B testing with real participants. We would like to clarify that the purpose of this study is to validate the judge's fidelity scores. Specifically, we test whether the judge's rankings agree with human assessments of how faithfully user simulators reproduce the original user behaviors.
To do so, we carefully constructed a validation set of 100 pairwise comparisons. Each example contains an original conversation and two simulated conversations, together with a preference that selects the more faithful simulated conversation. The authors selected these preferences through detailed analysis and discussion. We believe that it is necessary for the authors to do this task because it requires expertise and familiarity with the task and the behavior criteria in Appendix C. In particular, it requires distinguishing behavior fidelity from other metrics, such as general conversation quality (a more fluent simulation is not necessarily more faithful to the original user). During labeling, the simulations were presented in randomized order, with their source and scores hidden. Most importantly, we would like to highlight that the comparisons are grounded in observed user behavior, rather than the general impression of whether a conversation "looks human". Our validation therefore tests whether the judge's rankings agree with these carefully selected preferences. We will update Appendix B to be more clear about these details.
W3. Comparisons with Prior Work and Experiment Controls
The paper lacks validation on independent evaluation resources and comparisons with existing user-profile construction, behavioral representation, and specialized simulation methods and models.
We than the reviewer for suggesting these comparisons. We have conducted additional experiments with representations based on prior work, also reported in our responses to Reviewers eedj and uo1b:
- Task only: our task description, without a behavior description.
- Intent (UserLM): the user's overall goal, generated using UserLM's intent-generation prompt.
- Task + Persona (PersonaHub): our task description paired with a PersonaHub-style description of demographic and biographical attributes.
- Task + Implicit Profile (USP): our task description paired with an adapted USP profile that describes the user separately from the task.
We generate each representation with both Qwen3.5-4B and GPT-5.5. For the conversation reconstruction task, we report Best-of-1 / Best-of-3 fidelity:
| Generator | Task only | Intent (UserLM) | Task + Persona (PersonaHub) | Task + Implicit Profile (USP) | Task + Behavior (ours) |
|---|---|---|---|---|---|
| Qwen3.5-4B | 3.04 / 3.84 | 2.86 / 3.61 | 3.01 / 3.78 | 3.23 / 4.05 | 3.41 / 4.42 |
| GPT-5.5 | 3.16 / 3.98 | 3.04 / 3.77 | 3.09 / 3.84 | 3.51 / 4.39 | 3.89 / 4.90 |
| UserRepGen-4B | — | — | — | — | 4.95 / 6.02 |
We additionally report results for all three tasks in Section 7. For simulating users in new scenarios, we add PersonaHub-style personas and adapted USP profiles to the four conditions. For the added baselines, the persona or profile replaces the behavior description, and the task is held fixed across conditions:
| Generator | (A) | (B) | (C) | (D) |
|---|---|---|---|---|
| Task + Persona (PersonaHub) | 3.30 / 4.02 | 3.23 / 3.95 | 3.16 / 3.78 | 3.18 / 3.92 |
| Task + Implicit Profile (USP) | 3.30 / 4.02 | 3.32 / 4.03 | 3.49 / 4.20 | 3.57 / 4.45 |
| Qwen3.5-4B (base) | 3.30 / 4.02 | 3.46 / 4.16 | 3.37 / 4.17 | 3.80 / 4.68 |
| + SFT | 3.31 / 4.01 | 4.36 / 5.37 | 4.61 / 5.70 | 5.32 / 6.35 |
| + OPSD | 3.57 / 4.25 | 4.10 / 5.20 | 4.37 / 5.51 | 4.95 / 5.96 |
| + OSFT (UserRepGen-4B) | 3.43 / 4.16 | 4.48 / 5.50 | 4.69 / 5.80 | 5.45 / 6.59 |
For preserving the behavioral diversity at the populaton level, we include results for Intent, Task + Persona, and Task + Implicit Profile:
| Generator | Representation | JS | Self-BLEU |
|---|---|---|---|
| Qwen3.5-4B | Intent (UserLM) | 0.389 | 0.411 |
| Qwen3.5-4B | Task + Persona (PersonaHub) | 0.340 | 0.376 |
| Qwen3.5-4B | Task + Implicit Profile (USP) | 0.350 | 0.366 |
| Qwen3.5-4B (base) | Task + Behavior (ours) | 0.315 | 0.362 |
| + GRPO | Task + Behavior (ours) | 0.338 | 0.353 |
| + SFT | Task + Behavior (ours) | 0.270 | 0.319 |
| + OPSD | Task + Behavior (ours) | 0.308 | 0.333 |
| + OSFT (UserRepGen-4B) | Task + Behavior (ours) | 0.273 | 0.316 |
Rebuttal by Authors (3/n)
And finally, for distinguishing users across conversations, we replace the descriptions with personas or profiles in the same user-identification tasks:
| Generator | Representation | Accuracy |
|---|---|---|
| - | Full conversation (baseline) | 44.22 |
| Qwen3.5-4B | Persona (PersonaHub) | 34.47 |
| Qwen3.5-4B | Implicit Profile (USP) | 38.55 |
| Qwen3.5-4B | Behavior (ours) | 38.78 |
| + GRPO | Behavior (ours) | 41.50 |
| + SFT | Behavior (ours) | 43.54 |
| + OPSD | Behavior (ours) | 50.34 |
| + OSFT (UserRepGen-4B) | Behavior (ours) | 51.02 |
Across all four of these tasks, UserRepGen-4B consistently outperforms the added representation baselines. It leads to higher simulation fidelity for conversation reconstruction and for simulating a user in unseen scenarios, lower distributional gaps and lexical homogeneity at the population-level, and higher same-user identification accuracy.
Why we keep the simulation method and models fixed. In Section 3, we explain how recent works focus on optimizing the user simulator (i.e. the method or model), while representation generation is typically done with a general-purpose LLM. In our work, we focus on optimizing the user representation that the simulator will be conditioned on, and optimizing the model that generates these user representations. In our evaluation setup, we intentionally keep the simulator fixed so that we can isolate the effects of the representation. Since only the representation changes, and the simulator stays the same, it is a controlled setting where any differences in simulation fidelity can be attributed to the representation.
Lastly, we train using an ensemble of simulation environments based on Qwen3.5-4B and Gemma3-4B, and evaluate using GPT-5.4 mini. Using different models during training and evaluation is intended to reduce overoptimizing to a single simulator, and ensuring that the learned representations generalize to simulators not seen during training.
Q2. Robustness of JS divergence to clustering and finite sample size
Thank you for raising this question. Section 7.2 adopts the distributional gap metric introduced by [10], and follows the same configuration. [10] validates their evaluation approach by reporting sensitivity analyses across embedding models, clustering algorithms, clustering counts, and find that simulator rankings remain largely stable. Nevertheless, we agree that robustness should also be assessed in our setting. In particular, we use 700 conversations per group while the original work uses 5,000. We repeat the ablations for clustering counts to assess whether our results remain consistent across different cluster sizes. The table below reports Spearman correlations between the JS-divergence rankings at each cluster count and the default setting of k=500:
| Cluster count | 50 | 100 | 250 | 1000 |
|---|---|---|---|---|
| Spearman correlation with k=500 | 0.83 | 0.93 | 0.95 | 0.98 |
The rankings achieve strong correlations across the cluster counts, particularly for k >= 100. The lower correlation at k=50 indicates more sensitivity to coarse clustering, but the overall agreement supports the stability of the rankings across different cluster counts.
Rebuttal by Authors (4/n)
References
[1] Zhou, X., et al. (2026). OdysSim: Building foundation models for human behavior simulation. In The Fortieth Annual Conference on Neural Information Processing Systems.
[2] Liu, Z., et al. (2026). MUSE: Multi-domain Chinese user simulation via self-evolving profiles and rubric-guided alignment. In Findings of the Association for Computational Linguistics: EMNLP 2026.
[3] Wang, Y. S., et al. (2026). Learning user simulators with Turing rewards. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing.
[4] Wu, S., et al. (2026). HumanLM: Simulating users with state alignment beats response imitation. In Forty-third International Conference on Machine Learning.
[5] Wu, X., et al. (2026). Thinking alignment of scenario-oriented user simulation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
[6] Naous, T., et al. (2026). Flipping the dialogue: Training and evaluating user language models. In The Fourteenth International Conference on Learning Representations.
[7] Dou, Y., et al. (2025). SimulatorArena: Are user simulators reliable proxies for multi-turn evaluation of AI assistants? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.
[8] Wang, K., et al. (2025). Know you first and be you better: Modeling human-like user simulators via implicit profiles. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
[9] Wang, X., et al. (2025). CoSER: Coordinating LLM-based persona simulation of established roles. In Forty-second International Conference on Machine Learning.
[10] Ge, T., et al. (2024). Scaling synthetic data creation with 1,000,000,000 personas.
[11] Mehri, S., et al. (2026). Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors. In The Fortieth Annual Conference on Neural Information Processing Systems.
Thank you again for your constructive feedback. We hope that our additional experiments and explanations address the weaknesses and concerns that were raised. We would be happy discuss any remaining questions or concerns, and appreciate the time and effort the reviewer has put into our work.
Rebuttal by Authors (1/n)
Thank you for your careful review of our paper! We are encouraged that you recognize the importance of optimizing user representations and highlight that our paper is clearly and concisely written. We agree with your concerns regarding our evaluations, and report additional experiments and explanations in the sections below.
W1. Judge reliability and adaptation to the scoring rubric
We understand the reviewer's concern about GPT-5.4 mini's ability to judge behavior fidelity. We put extensive effort into carefully designing our evaluation to ensure that it measures realism well. Firstly, we developed the criteria and behavioral dimensions that were important to this task, and examined whether the judge's rationales and scores reflected the intended comparisons. As described in Appendix C, these dimensions cover communication style, engagement, request formulation, responses to assistant outputs, and context provision. Importantly, our evaluation compares simulated behavior with the original user behavior, rather than simply asking whether a conversation appears plausible.
We would also like to mention that evaluating user simulators with rubrics and LLM-judges is a well-established practiced for evaluating realism and behavioral fidelity in user simulation research:
Nevertheless, we share concerns with the reviewer regarding only relying on an LLM judge. For this reason, we complement it with two evaluations that do not use the the judge's fidelity score:
We agree with the reviewer, and re-evaluated the conversations with four additional judges: Qwen3.5-122B-A10B, gpt-oss-120b, GPT-6 Luna, and GPT-6 Sol. These are also reported in our responses to the other reviewers. We report Best-of-1 / Best-of-3 fidelity:
Our conclusions hold under every judge, and the relative orderings of the representation generators are nearly identical. The table below reports Spearman rank correlations with the original GPT-5.4 mini judge: