- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MIMESIS-9B, a user simulation model trained using interaction data between real users and LLMs. The paper describes an approach to train this model based on a midtraining phase then a post-training phase with reasoning data. The paper then evaluates the model on previous benchmarks for user simulation. Finally, the simulator model is used to train an LLM via multi-turn RL comparing the use of GPT5.5 and MIMESIS-9B as the simulator, with also a modification for the GRPO objective.
Critical strengths
-
One of the interesting issues that this paper raises is the ability of models to generalize to new users beyond simulators that it has seen in training. I feel this is an important direction for training LLMs with simulators which has seen less focus in past literature. Although, this part of the paper comes only at the end with limited experimental results.
-
I also find splitting the training of the simulator into a “midtraining” part on interaction data between users and LLMs, then a post-training part that introduces reasoning into the model.
Critical weaknesses
-
The first bullet point on page 3 of the introduction is inaccurate as a novel contribution of this paper (showing how there is a behavioral mismatch between simulators based on off-the-shelf LLMs and real humans) since this has been extensively shown in several recent papers.
-
The related work is poorly written: it does not accurately summarize the recent literature on user simulation nor show how this work is different from the most closely related papers on this specific topic. Parts of the related work also feel not directly relevant to the problem the paper is addressing. For example, the first part discusses the history of benchmarks developed for evaluating agentic capabilities of LLMs.
-
The description of the training approach of the simulator and what training data is used is not very clear. For example, in Section 4.2, the paper briefly refers to using a joint reinforcement learning approach that was adopted from a past paper, but does not describe things in a clear way, making it difficult to understand the method used in this paper, and what is particularly new. As far as I understand, this model is trained to perform multiple tasks beyond only chatting as a user (e.g. as the paper mentioned “Cognition” and “Social Simulation”). It’s not clear what the distinction for these things are, whether and why this is a valid/useful approach compared to chat-based simulators, which are more aligned with the end goal of training agents in open-domain chat.
-
The paper jumps directly in Section 4.4 to simulator evaluation using 4 different datasets without giving clear descriptions of these resources or what they are evaluating (although it only shows the results on one benchmark in Table 1). It is difficult to interpret the numbers in Table 1 without more context about the datasets. It seems that more details and results are provided in the Appendix, but such critical information for paper understanding, especially since they are main results, should have been in the main paper. In my opinion, the paper uses a lot of space for unnecessary math notations (such as the equation for GRPO on line 385), which could have been used for more important content.
-
I find similar issues related to lack of clarity in Section 5 that uses the simulator for RL training of an LLM agent (no details on the dataset used for this setup, what the datasets the evaluation reported in Table 2 are on, etc.). Further, the results in Table 2 seem to be based on a single run and would benefit from additional runs and statistical significance testing. These experiments are also missing an important comparison to training with other trained simulators besides MIMESIS-9B, and open-source prompted simulators where the modified training objective can be applied.
Questions for authors
Please see the weaknesses section which raises some questions and provides recommendations to strengthen the paper and results.
3 comments
Response to Reviewer gYmk (2/2)
Weakness 4 — Benchmarks not described; only one benchmark in Table 1; unnecessary math
The main text already reports all four benchmarks in Fig. 1. To make them easier to interpret, the updated PDF adds the definitions below and a summary table (Table R1) to Sec. 4.4.
| Benchmark | What it measures | Better |
|---|---|---|
| SOUL | Simulation capability on five axes (conversation, social, cognition, role play, judgment), on held-out instances | ↑ |
| RealUserSim PT3 | Agreement with 600 human WildChat trajectories (persona, style, competence, information flow, pacing) | ↑ |
| τ-USI | Behavioral alignment with human τ-bench users and calibration of task outcomes | ↑ |
| SimulatorArena | Turing distance: how well a judge separates simulated from human conversations | ↓ |
| PRISM | Open-domain next-turn realism against human users (128 users, 880 turns) | ↑ |
To make Table 1 easier to interpret, the table below lists the benchmarks in each SOUL axis and the axis scores of key models (GPT-5.6-Sol judge, Claude-Opus-5.5 partner). Bold marks the best score in each row among all models in Table 1. MIMESIS-9B outperforms the same-backbone Osim3.5-9B on every axis and is within 0.5 points of the best COG score (86.7, GPT-5.5), whereas Claude-Opus-5 remains the strongest model on SS, ROLE and EVAL.
| SOUL axis (benchmarks) | MIMESIS-9B | MIMESIS-4B | Claude-Opus-5 | Osim3.5-9B | OSim-8B | Qwen3.5-9B |
|---|---|---|---|---|---|---|
| CONV (UserLLM, MirrorBench, Humanual-Chat, SimArena-Doc) | 79.8 | 80.0 | 64.2 | 65.0 | 65.6 | 53.4 |
| SS (Sotopia-Hard) | 66.9 | 65.8 | 72.8 | 43.8 | 42.1 | 56.8 |
| COG (Fantom, Hitom, Paratomi, Social-R1) | 86.2 | 80.4 | 82.5 | 81.3 | 74.2 | 63.9 |
| ROLE (CoSER, Lifechoices, Twinvoice, BehaviorChain, SimArena-Math, Mistakes, Humanual-Email/-News/-Politics) | 67.5 | 63.9 | 71.3 | 59.3 | 58.9 | 50.9 |
| EVAL (AlignX, SocSci210, HumanLLM, Humanual-Book/-Opinion) | 61.9 | 60.1 | 63.8 | 54.9 | 56.7 | 49.2 |
| Overall (mean of 23 benchmarks) | 71.6 | 68.8 | 70.5 | 62.5 | 61.5 | 53.5 |
Unnecessary math. We keep the GRPO equations for two reasons. First, they are the only place where the paper writes out GRPO, so they keep the paper self-contained for readers new to it. Second, CSD is defined against them, in the same notation. With the 10-page limit for the revision, we believe the main paper now has enough space for all necessary information, including the benchmark definitions above.
Weakness 5 — Agent setup, single runs, and missing simulator baselines
Tasks and data Agents are trained in five gyms from UserRL and evaluated on three further held-out gyms:
| Gym | What the agent must do | Split |
|---|---|---|
| Travel | Elicit preferences for travel planning | Train |
| Turtle | Solve lateral-thinking puzzles | Train |
| Function | Infer a hidden function by querying the user | Train |
| Tau | τ-bench customer service | Train |
| Persuade | Change a user's stance | Train |
| Intention | Clarify vague requests | Held out |
| Telepathy | Guess what the user has in mind | Held out |
| Search | Answer a user's information need with search | Held out |
For evaluation, we use nine user models, none of which is used to train the agents in Table 2. Four of them are released simulators (Ditto-8B, OSim-8B, HumanLM-8B, Sotopia-7B), and five are frontier models.
Multiple runs and significance. To address the concern about single runs, all results now use five training seeds with 95% CIs and paired tests (Table R2). Importantly, the simulator effect (+3.52) and the CSD effect (+2.06) are each positive under every evaluation user, with p < 0.001.
Other trained and open-source prompted simulators. Using the same agent and settings, with five seeds, we train GRPO, InfoPO, and CSD with four other training users: the learned simulators OSim-8B and Ditto-8B, and the prompted open-weight models DeepSeek-V4.1-Flash and Kimi-K3 (already reported in our original submission). CSD achieves the highest training and validation reward under all four, which shows that the modified objective transfers to other simulators. In addition, when evaluated by the other learned simulators themselves (OSim-8B, Ditto-8B, HumanLM-8B, Sotopia-7B), the MIMESIS-trained agent outperforms GPT-5.5-based training.
At step 100, GRPO and CSD reach the following rewards (means over five seeds; relative gain in parentheses):
| DeepSeek-V4.1-Flash | Kimi-K3 | OSim-8B | Ditto-8B | |
|---|---|---|---|---|
| Training | 0.84 → 1.02 (+20%) | 0.95 → 1.55 (+63%) | 1.65 → 2.41 (+46%) | 1.82 → 2.66 (+46%) |
| Validation | 0.54 → 0.69 (+29%) | 0.55 → 1.09 (+99%) | 1.05 → 1.47 (+40%) | 1.10 → 1.42 (+29%) |
We thank the reviewer again for these suggestions, which have made the paper clearer and stronger.
Follow-up on our response
Dear Reviewer gYmk,
We wanted to follow up on our response, in particular the multi-seed agent experiments and the additional training simulators you asked for. Please let us know if any concerns remain, so that we have time to address them before the discussion ends.
Best regards, Authors of Submission 35581
Response to Reviewer gYmk (1/2)
We thank the reviewer for the constructive review, below we address each weakness in turn.
We agree that the existence of the mismatch is known, and we cite UserLM and τ-USI accordingly. Our contribution is to quantify its consequence for agent training and to close it. Prior work established the mismatch for evaluation, or for training chat assistants. However, we show that even trained simulators still have a large gap to humans: OSim-8B and Osim3.5-9B fall 10.3 and 13.9 pp below the human rate, as do all specialized simulators evaluated in τ-USI. They only learn to imitate observable user behavior via SFT on human chat, so they mimic how users talk but not what users think and also fail to maintain realistic behaviors in long-horizon interactions. MIMESIS also learns from users' self-reported thoughts and matches the human rate within 2.5 pp; without this supervision, its reasoning collapses. We have therefore rephrased the bullet to state that we quantify the mismatch and its consequence for agent training rather than identify it.
We initially omitted a comparison from the related work because we discussed the baseline simulators in the experimental setup. To address this concern, Sec. 2 now includes a dedicated paragraph on learned user simulators (p. 4). It contrasts MIMESIS with UserLM, which models user turns; OdysSim, which uses task-specific RL and distillation; Ditto, which relies on feedback-refined rollouts; HumanLM and UserLM-R1, which model latent user states; and Turing-RL. Unlike these approaches, MIMESIS supervises the simulator's private reasoning with users' own self-reports and explicitly rewards grounded, realistic behaviors. In addition, the updated PDF condenses the agent-benchmark history that the reviewer found less relevant and adds an explicit contrast with Suh et al.
Joint RL. Joint RL on this task suite is part of our recipe and is not adopted from prior work. OdysSim, which introduced the 23 SOUL tasks, found that "directly mixing all tasks into a single RL run is suboptimal" and therefore trains a separate RL expert per task and then distills the experts into one model. In contrast, we update a single simulator with rollouts from all 24 environments, each with its own reward - which is much more convenient and efficient. This works because RL starts from our mid-trained checkpoint, which keeps validation reward stable across most tasks and lets RL keep improving compared to the original OdysSim recipe. Ditto also trains a single model, but on a smaller suite of 10 tasks and starting from an instruction-tuned model rather than a user-side mid-trained one.
What is new. The new elements are (i) thought supervision from users' self-reports and (ii) the realistic-behavior objective. Figs. 11 and 13 isolate the effect of both, and on the same backbone the full recipe outperforms OdysSim's by 9 SOUL points.
Why non-chat tasks? Within SOUL itself, MIMESIS's largest advantage is on conversational simulation (CONV 79.8, while every baseline scores at most 67.2), which is the core capability of a user simulator. Furthermore, to train agents, a simulator must act like a user, not only talk like one. Real users pursue goals they do not state, keep track of what the agent knows, act consistently with who they are, and judge answers by their own standards. Chat logs show these aspects only indirectly, so each RL task trains one of them directly: