acceptodds
Under review as a conference paper at ICLR 2027

MIMESIS: Learning User Simulators as Training Environments for Interactive Agents

59 views

Abstract

Training and evaluating interactive language agents typically requires rich user interactions, yet collecting human feedback is expensive and difficult to scale. Simulated users offer a scalable alternative, but they must both resemble real user behavior and provide useful learning experiences for agents. In contrast, most agent-training frameworks rely on off-the-shelf assistant LLMs, whose helpfulness can make them overly cooperative, explicit, and behaviorally homogeneous compared with real users. We introduce MIMESIS, a purpose-built user simulator trained on human conversations with explicit reasoning supervision and 13 realistic behavioral patterns derived from real user interactions. Empirically, our 9B model achieves a SOUL-Index of 65.7, surpassing the strongest frontier model. Compared with Opus 5.0, the strongest baseline on RealUserSim and SimulatorArena, MIMESIS improves behavioral fidelity by 13.4 points and reduces Turing distance by 3.6 points, respectively. We then freeze the simulator and train an agent through interaction with it with multi-turn reinforcement learning. Across eight environments, training with MIMESIS yields better agent performance than training with GPT-5.5 under all nine unseen user simulators, demonstrating stronger generalization to new users. Moreover, we propose Coached On-Policy Self-Distillation (CSD), which leverages the simulator’s internal thoughts and subsequent utterances as feedback on how well the agent addresses user needs. A coach converts this information into concise coaching notes that describe how the agent can better anticipate user needs and adapt its behavior over the course of an interaction. CSD turns this feedback into dense, token-level supervision beyond sparse task rewards, yielding further gains across all nine evaluation user models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
user simulator

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gYmkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MIMESIS-9B, a user simulation model trained using interaction data between real users and LLMs. The paper describes an approach to train this model based on a midtraining phase then a post-training phase with reasoning data. The paper then evaluates the model on previous benchmarks for user simulation. Finally, the simulator model is used to train an LLM via multi-turn RL comparing the use of GPT5.5 and MIMESIS-9B as the simulator, with also a modification for the GRPO objective.

Critical strengths
  • One of the interesting issues that this paper raises is the ability of models to generalize to new users beyond simulators that it has seen in training. I feel this is an important direction for training LLMs with simulators which has seen less focus in past literature. Although, this part of the paper comes only at the end with limited experimental results.

  • I also find splitting the training of the simulator into a “midtraining” part on interaction data between users and LLMs, then a post-training part that introduces reasoning into the model.

Critical weaknesses
  • The first bullet point on page 3 of the introduction is inaccurate as a novel contribution of this paper (showing how there is a behavioral mismatch between simulators based on off-the-shelf LLMs and real humans) since this has been extensively shown in several recent papers.

  • The related work is poorly written: it does not accurately summarize the recent literature on user simulation nor show how this work is different from the most closely related papers on this specific topic. Parts of the related work also feel not directly relevant to the problem the paper is addressing. For example, the first part discusses the history of benchmarks developed for evaluating agentic capabilities of LLMs.

  • The description of the training approach of the simulator and what training data is used is not very clear. For example, in Section 4.2, the paper briefly refers to using a joint reinforcement learning approach that was adopted from a past paper, but does not describe things in a clear way, making it difficult to understand the method used in this paper, and what is particularly new. As far as I understand, this model is trained to perform multiple tasks beyond only chatting as a user (e.g. as the paper mentioned “Cognition” and “Social Simulation”). It’s not clear what the distinction for these things are, whether and why this is a valid/useful approach compared to chat-based simulators, which are more aligned with the end goal of training agents in open-domain chat.

  • The paper jumps directly in Section 4.4 to simulator evaluation using 4 different datasets without giving clear descriptions of these resources or what they are evaluating (although it only shows the results on one benchmark in Table 1). It is difficult to interpret the numbers in Table 1 without more context about the datasets. It seems that more details and results are provided in the Appendix, but such critical information for paper understanding, especially since they are main results, should have been in the main paper. In my opinion, the paper uses a lot of space for unnecessary math notations (such as the equation for GRPO on line 385), which could have been used for more important content.

  • I find similar issues related to lack of clarity in Section 5 that uses the simulator for RL training of an LLM agent (no details on the dataset used for this setup, what the datasets the evaluation reported in Table 2 are on, etc.). Further, the results in Table 2 seem to be based on a single run and would benefit from additional runs and statistical significance testing. These experiments are also missing an important comparison to training with other trained simulators besides MIMESIS-9B, and open-source prompted simulators where the modified training objective can be applied.

Questions for authors

Please see the weaknesses section which raises some questions and provides recommendations to strengthen the paper and results.

3 comments
AuthorsOpenReview ↗

Response to Reviewer gYmk (1/2)

We thank the reviewer for the constructive review, below we address each weakness in turn.

Weakness 1 — The first contribution bullet (behavioral mismatch)

We agree that the existence of the mismatch is known, and we cite UserLM and τ-USI accordingly. Our contribution is to quantify its consequence for agent training and to close it. Prior work established the mismatch for evaluation, or for training chat assistants. However, we show that even trained simulators still have a large gap to humans: OSim-8B and Osim3.5-9B fall 10.3 and 13.9 pp below the human rate, as do all specialized simulators evaluated in τ-USI. They only learn to imitate observable user behavior via SFT on human chat, so they mimic how users talk but not what users think and also fail to maintain realistic behaviors in long-horizon interactions. MIMESIS also learns from users' self-reported thoughts and matches the human rate within 2.5 pp; without this supervision, its reasoning collapses. We have therefore rephrased the bullet to state that we quantify the mismatch and its consequence for agent training rather than identify it.

Weakness 2 — Related work

We initially omitted a comparison from the related work because we discussed the baseline simulators in the experimental setup. To address this concern, Sec. 2 now includes a dedicated paragraph on learned user simulators (p. 4). It contrasts MIMESIS with UserLM, which models user turns; OdysSim, which uses task-specific RL and distillation; Ditto, which relies on feedback-refined rollouts; HumanLM and UserLM-R1, which model latent user states; and Turing-RL. Unlike these approaches, MIMESIS supervises the simulator's private reasoning with users' own self-reports and explicitly rewards grounded, realistic behaviors. In addition, the updated PDF condenses the agent-benchmark history that the reviewer found less relevant and adds an explicit contrast with Suh et al.

Weakness 3 — Training approach and data; what is new; why non-chat tasks

StageDataTraining signal
User-side mid-training21.2M examples from 62 corpora; 9.51B tokensPredict the next human utterance from the preceding dialogue
ThoughtTrace SFT2,155 conversations with users' self-reported thoughtsPrivate reasoning, then the observed utterance
Joint RL23 SOUL tasks + the ABCD behavior taskTask-specific rewards; all rollouts update one shared model

Joint RL. Joint RL on this task suite is part of our recipe and is not adopted from prior work. OdysSim, which introduced the 23 SOUL tasks, found that "directly mixing all tasks into a single RL run is suboptimal" and therefore trains a separate RL expert per task and then distills the experts into one model. In contrast, we update a single simulator with rollouts from all 24 environments, each with its own reward - which is much more convenient and efficient. This works because RL starts from our mid-trained checkpoint, which keeps validation reward stable across most tasks and lets RL keep improving compared to the original OdysSim recipe. Ditto also trains a single model, but on a smaller suite of 10 tasks and starting from an instruction-tuned model rather than a user-side mid-trained one.

What is new. The new elements are (i) thought supervision from users' self-reports and (ii) the realistic-behavior objective. Figs. 11 and 13 isolate the effect of both, and on the same backbone the full recipe outperforms OdysSim's by 9 SOUL points.

Why non-chat tasks? Within SOUL itself, MIMESIS's largest advantage is on conversational simulation (CONV 79.8, while every baseline scores at most 67.2), which is the core capability of a user simulator. Furthermore, to train agents, a simulator must act like a user, not only talk like one. Real users pursue goals they do not state, keep track of what the agent knows, act consistently with who they are, and judge answers by their own standards. Chat logs show these aspects only indirectly, so each RL task trains one of them directly:

AxisWhat the simulator does
CONVWrites the next user turn in real human–LLM chats and document-writing sessions
SSPursues a private goal while negotiating, collaborating or arguing with another agent
COGAnswers who knows what in multi-party conversations and in nested- and false-belief stories
ROLEStays in a character, persona, student or real-user role, and predicts that person's next choice, action or text, or a typical student error
EVALPredicts a specific person's preferences, survey ratings and reviews from their profile
ABCD realistic behaviorsPlays a customer-support user who shows one of 13 behaviors in a grounded scenario
AuthorsOpenReview ↗

Response to Reviewer gYmk (2/2)

Weakness 4 — Benchmarks not described; only one benchmark in Table 1; unnecessary math

The main text already reports all four benchmarks in Fig. 1. To make them easier to interpret, the updated PDF adds the definitions below and a summary table (Table R1) to Sec. 4.4.

BenchmarkWhat it measuresBetter
SOULSimulation capability on five axes (conversation, social, cognition, role play, judgment), on held-out instances↑
RealUserSim PT3Agreement with 600 human WildChat trajectories (persona, style, competence, information flow, pacing)↑
τ-USIBehavioral alignment with human τ-bench users and calibration of task outcomes↑
SimulatorArenaTuring distance: how well a judge separates simulated from human conversations↓
PRISMOpen-domain next-turn realism against human users (128 users, 880 turns)↑

To make Table 1 easier to interpret, the table below lists the benchmarks in each SOUL axis and the axis scores of key models (GPT-5.6-Sol judge, Claude-Opus-5.5 partner). Bold marks the best score in each row among all models in Table 1. MIMESIS-9B outperforms the same-backbone Osim3.5-9B on every axis and is within 0.5 points of the best COG score (86.7, GPT-5.5), whereas Claude-Opus-5 remains the strongest model on SS, ROLE and EVAL.

SOUL axis (benchmarks)MIMESIS-9BMIMESIS-4BClaude-Opus-5Osim3.5-9BOSim-8BQwen3.5-9B
CONV (UserLLM, MirrorBench, Humanual-Chat, SimArena-Doc)79.880.064.265.065.653.4
SS (Sotopia-Hard)66.965.872.843.842.156.8
COG (Fantom, Hitom, Paratomi, Social-R1)86.280.482.581.374.263.9
ROLE (CoSER, Lifechoices, Twinvoice, BehaviorChain, SimArena-Math, Mistakes, Humanual-Email/-News/-Politics)67.563.971.359.358.950.9
EVAL (AlignX, SocSci210, HumanLLM, Humanual-Book/-Opinion)61.960.163.854.956.749.2
Overall (mean of 23 benchmarks)71.668.870.562.561.553.5

Unnecessary math. We keep the GRPO equations for two reasons. First, they are the only place where the paper writes out GRPO, so they keep the paper self-contained for readers new to it. Second, CSD is defined against them, in the same notation. With the 10-page limit for the revision, we believe the main paper now has enough space for all necessary information, including the benchmark definitions above.

Weakness 5 — Agent setup, single runs, and missing simulator baselines

Tasks and data Agents are trained in five gyms from UserRL and evaluated on three further held-out gyms:

GymWhat the agent must doSplit
TravelElicit preferences for travel planningTrain
TurtleSolve lateral-thinking puzzlesTrain
FunctionInfer a hidden function by querying the userTrain
Tauτ-bench customer serviceTrain
PersuadeChange a user's stanceTrain
IntentionClarify vague requestsHeld out
TelepathyGuess what the user has in mindHeld out
SearchAnswer a user's information need with searchHeld out

For evaluation, we use nine user models, none of which is used to train the agents in Table 2. Four of them are released simulators (Ditto-8B, OSim-8B, HumanLM-8B, Sotopia-7B), and five are frontier models.

Multiple runs and significance. To address the concern about single runs, all results now use five training seeds with 95% CIs and paired tests (Table R2). Importantly, the simulator effect (+3.52) and the CSD effect (+2.06) are each positive under every evaluation user, with p < 0.001.

Other trained and open-source prompted simulators. Using the same agent and settings, with five seeds, we train GRPO, InfoPO, and CSD with four other training users: the learned simulators OSim-8B and Ditto-8B, and the prompted open-weight models DeepSeek-V4.1-Flash and Kimi-K3 (already reported in our original submission). CSD achieves the highest training and validation reward under all four, which shows that the modified objective transfers to other simulators. In addition, when evaluated by the other learned simulators themselves (OSim-8B, Ditto-8B, HumanLM-8B, Sotopia-7B), the MIMESIS-trained agent outperforms GPT-5.5-based training.

At step 100, GRPO and CSD reach the following rewards (means over five seeds; relative gain in parentheses):

DeepSeek-V4.1-FlashKimi-K3OSim-8BDitto-8B
Training0.84 → 1.02 (+20%)0.95 → 1.55 (+63%)1.65 → 2.41 (+46%)1.82 → 2.66 (+46%)
Validation0.54 → 0.69 (+29%)0.55 → 1.09 (+99%)1.05 → 1.47 (+40%)1.10 → 1.42 (+29%)

We thank the reviewer again for these suggestions, which have made the paper clearer and stronger.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer gYmk,

We wanted to follow up on our response, in particular the multi-seed agent experiments and the additional training simulators you asked for. Please let us know if any concerns remain, so that we have time to address them before the discussion ends.

Best regards, Authors of Submission 35581

Reviewer_wcopOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MIMESIS, a user simulator for training interactive agents. The authors train the simulator using human conversations, explicit reasoning supervision, and joint multi-domain RL with 13 realistic user behaviors. They also propose CSD, which uses feedback from the trained user simulator to provide additional supervision for agent training. The authors evaluate MIMESIS on various user simulation benchmarks and report improvements over existing baselines. They further show that training agents with MIMESIS improves performance across unseen user simulators.

Critical strengths

I think training realistic user simulators is an very important research direction. The paper includes a wide range of experiments across different benchmarks and settings, making the empirical evaluation comprehensive. I think the overall flow of the paper is well organized.

Critical weaknesses

The performance improvement over OdysSim seems strange to me in two aspects, and because of that I am not sure whether there is fundamental novelty in this work.

  1. Since the authors use Qwen3.5-9B as the base model for training, isn't it natural that it performs better? It is not clear whether training with the method proposed in MIMESIS actually makes a meaningful difference. I think the paper should report the performance of Qwen3.5-9B trained with the OdysSim method, or train Qwen3-8B using the MIMESIS method, to show that this is actually a meaningful contribution.

  2. OdysSim uses GPT-5-nano as both the reward model and the judge in the original paper, but this paper uses GPT-5.5 as the judge, which seems to result in a performance drop for OdysSim compared to the original paper. Moreover, since the reward model used for training MIMESIS is also GPT-5.5, MIMESIS seems to have a significant advantage from the beginning. I think this requires further explanation or ablation experiments.

Also, as far as I remember, the OdysSim paper claimed that directly training a single model with joint multi-task RL does not work well, which is why it adopted task-specific RL followed by expert distillation. However, this paper uses joint RL without clearly explaining why this approach would work better here. Although MIMESIS additionally introduces reasoning supervision, realistic behavior training, and CSD, I am still not sure what the fundamental novelty is beyond using an approach that OdysSim already considered and found less effective.

Questions for authors

Could you please justify for the 1, 2 mentioned in Critical Weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer wcop (1/2)

We thank the reviewer for the clear and actionable questions. We ran exactly the controls requested in Weaknesses 1 and 2 and address joint RL and novelty under Weakness 3.

Weakness 1 / Question — "Isn't it natural that [Qwen3.5-9B] performs better?"

On the same backbone, MIMESIS outperforms the OdysSim recipe by a wide margin on every external benchmark and on 21 of 23 SOUL benchmarks. As suggested, we trained Osim3.5-9B by applying the OdysSim recipe to Qwen3.5-9B, the backbone of MIMESIS-9B. We report the results and then highlight the main differences between the two training recipes and how each affects final performance below.

SimulatorBackboneSOUL ↑ (two partners)RealUserSim ↑τ-USI ↑SimulatorArena ↓τ-bench gap to humans (pp) ↓
Qwen3-8B (pretrained)—51.8 / 51.444.460.249.318.4
OSim-8BQwen3-8B61.5 / 61.457.780.448.710.3
Qwen3.5-9B (pretrained)—53.5 / 53.939.457.846.314.9
Osim3.5-9BQwen3.5-9B62.5 / 62.262.365.348.113.9
MIMESIS-9BQwen3.5-9B71.6 / 71.294.080.238.72.4
Qwen3.5-4B (pretrained)—48.1 / 47.537.160.847.716.0
MIMESIS-4BQwen3.5-4B68.8 / 68.889.778.042.01.8

On the same backbone, MIMESIS-9B improves SOUL by +9.1 / +9.0 points, with the lead holding under both partners. It also gains +31.7 on RealUserSim and +14.9 on τ-USI, reduces the Turing distance by 9.4, and narrows the success-rate gap to humans from 13.9 to 2.4 pp. In contrast, swapping the backbone alone moves the OdysSim recipe by only +1.0 / +0.8 SOUL points. Notably, applying the OdysSim recipe ourselves does not reproduce the τ-USI score of the released OSim-8B (65.3 vs. 80.4), and our own training procedure reaches that level only once the realistic-behavior RL objective is added (80.2 vs. 73.9 without it).

MIMESIS-4B is built on Qwen3.5-4B, which before training is weaker than Qwen3-8B, the backbone of OSim-8B, on SOUL and RealUserSim (table above). Nevertheless, MIMESIS-4B outperforms the released OSim-8B on both, and also on SimulatorArena.

Weakness 2 / Question — GPT-5.5 as both reward model and judge; OdysSim's lower numbers

The ranking holds with a judge different from the reward model. To test for reward-model bias, we now judge all models with GPT-5.6-Sol under two interaction partners, and MIMESIS-9B ranks first under both. On the same 23-benchmark mean, MIMESIS-9B leads OSim-8B by +8.9 under the original GPT-5.5 judge and by +10.1 / +9.8 under GPT-5.6-Sol. Beyond SOUL, judges from three providers (GPT-5.6, Claude-Opus-5, Gemini-3.8-Flash) prefer MIMESIS over every baseline in next-turn realism on PRISM. Importantly, under every judge and interaction partner we used, MIMESIS leads OSim-8B by about 9–10 points. On PRISM, MIMESIS wins more often than it loses in all 24 judge–baseline comparisons.

AuthorsOpenReview ↗

Response to Reviewer wcop (2/2)

Weakness 3 — Joint RL (which OdysSim found less effective) and novelty

Joint RL is effective in our recipe because it starts from a good mid-trained checkpoint. As Fig. 9 shows, the initialization determines whether joint RL is stable. From the base model, validation reward degrades on many tasks, consistent with OdysSim's observation. In contrast, from our mid-trained checkpoint, validation reward stays higher and stable across most tasks. Moreover, on the same backbone, the jointly trained MIMESIS-9B outperforms the expert-distilled Osim3.5-9B on 21 of 23 benchmarks. Ditto likewise trains a single model jointly.

In our revised manuscript, we further find that a stronger mid-training stage is what allows RL to keep improving (Fig. 10). With identical RL, a model mid-trained at learning rate 1e-5 (the setting used by OdysSim) peaks at 49.4 and then declines, whereas mid-training at 3e-5 reaches 59.7. Thus, we find that the key is to climb as far as possible with mid-training before relying on expensive RL.

Mid-training LRTraining loss (start → end)RL score at step 0RL score at step 400
1×10⁻⁵3.45 → 2.2236.69.5
3×10⁻⁵3.43 → 1.2134.359.7

Beyond these findings, our work makes three contributions:

  • (i) Explicit reasoning supervised by human thoughts. A simulator must capture why users respond as they do, not only what they say, this also help the assistant understand what they need. Prior user simulators either imitate observable utterances only (UserLM, OSim, Ditto, Suh et al.) or model latent user states with reasoning inherited from assistant training (HumanLM, UserLM-R1), which does not reflect how humans actually think. Supervising the simulator's reasoning with users' own self-reported thoughts makes it think like a user rather than like an assistant, and these thoughts are what CSD later turns into feedback for the agent. Without ThoughtTrace, user-side training collapses the simulator's reasoning into one-word thoughts (e.g., MISTAKES 36.0). With it, the simulator reasons fully and reaches 63.0, surpassing even the pretrained backbone (60.7; Fig. 11).
  • (ii) A realistic-behavior RL objective. Real users often hide their criteria, shift goals, or contradict themselves, whereas assistant-style simulators are overly cooperative and inflate agent success by about 20 pp on τ-bench (Table R1). An agent trained against such users never learns to handle these behaviors, so we reward grounded, realistic behaviors explicitly. With everything else fixed, it improves RealUserSim (+1.7), τ-USI (+6.3), and SimulatorArena (−2.3 Turing distance) (Fig. 13), while cooperativeness on neutral scenarios stays at 0.98–0.99 (Fig. 14).
  • (iii) CSD. A learned simulator knows more than it says: its private thoughts reveal why an agent's turn helped or failed. Standard RL discards this information and uses only the scalar task reward, which is sparse and gives no signal when all rollouts in a group receive the same reward. CSD turns the simulator's private thoughts into token-level supervision, which helps the agent anticipate users' unspoken needs. With the same simulator, it adds +2.06 over GRPO under all nine users, whereas generic or mismatched notes add approximately zero.

We hope this resolves the reviewer's concerns and welcome any further questions.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer wcop,

We ran the some controlled experiments you suggested and would be grateful to hear whether the new results address your concerns. If any questions remain, we would be glad to answer them while the discussion period is still open.

Best regards, Authors of Submission 35581

Reviewer_puknOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes to learn a purpose-built user simulator and then use it as the training environment for interactive agents. It first argues that off-the-shelf assistant LLMs distort task difficulty when used as simulated users, and Figure 2 reports that a fixed GPT-5.5 agent on τ-bench behaves differently under real users and under LLM simulators. The authors then mid-train 4B and 9B simulators on a large human-conversation mixture, add ThoughtTrace supervision of users' self-reported thoughts, and apply joint multi-domain RL over SOUL tasks and an ABCD customer-support task with 13 behavior categories. The frozen simulator is used to train a Qwen3-8B agent with multi-turn GRPO in Gym environments, and the paper introduces Coached On-Policy Self-Distillation, which converts the simulator's private thought and next utterance into a coaching note that conditions a self-teacher and gates an auxiliary token-level likelihood loss. Simulator quality is measured on SOUL-Index, RealUserSim PT3, τ-USI, SimulatorArena and PRISM, and agent quality is measured with nine user models over held-in and held-out Gym environments. In the headline results the simulator is slightly above the strongest frontier model on SOUL-Index and substantially above it on RealUserSim, and the agent mean score rises when GPT-5.5 is replaced by the learned simulator and again when CSD is added.

Critical strengths

The learned simulator is clearly stronger than off-the-shelf baselines on external benchmarks. On RealUserSim PT3 it reaches a substantially higher fidelity index than Claude-Opus-5, on SimulatorArena it has the lowest Turing distance, on τ-USI it beats GPT-5.5 and is close to the strongest released simulator, and on PRISM it is statistically tied with the best frontier model. The reasoning-supervision and realistic-behavior ablations also show consistent gains on external benchmarks. This is the paper's strongest evidence that its training recipe produces a more human-like simulator.

The controlled simulator-swap experiment is the right test of the thesis, and its direction is robust. Replacing only the training simulator, while holding the agent, the GRPO objective, and the budget fixed, improves agent performance under all nine evaluation users. We independently re-aggregated the per-environment scores and found that the improvement also holds under an unweighted mean over environments and that mean rank improves. This is a meaningful result and the reason a weak rejection rather than a clear rejection is appropriate.

Critical weaknesses

The headline SOUL-Index result is not a held-out evaluation, and the index cannot be reconstructed. The simulator is trained with RL on SOUL tasks using GPT-5.5 as the reward and judge model, and Figure 9 logs validation reward on exactly the task family that SOUL-Index reports, with Table 7 evaluating with GPT-5.5 as assistant and judge where required. OdysSim, which introduces SOUL, describes the benchmark as the same task suite named as the simulator's RL mixture in Table 6. No held-out split, judge-independence check, or no-SOUL-RL ablation is described. As a result, the paper's lead over Claude-Opus-5 is less than a point while frontier models still lead the social-simulation and role-play axes, and the comparison is trained-versus-zero-shot rather than like-for-like. The index itself is not verifiable: Table 1's EVAL axis uses only part of the EVAL rows listed in Table 7, Table 7 prints more rows than its caption count, and the overall score cannot be derived from either the five axes or the dataset rows. A second problem affects the largest external margin. The mid-training mixture is not listed, RealUserSim PT3 is built from WildChat conversations, PRISM is a public human-chat corpus, and τ-USI uses τ-bench human turns; without decontamination the RealUserSim gain cannot be interpreted as human fidelity.

The agent-training evidence is single-run, uses an undocumented aggregate, includes an invalid evaluation block, and is not positioned against prior work. No seed count, confidence interval, or statistical test is reported for any agent result, and Figures 5 and 13 show one curve per method. The "Avg." column is not an average over the eight environments; the reported values can be reproduced only by an undocumented weighting dominated by one held-in environment, and the per-environment scores show that several conclusions do not hold under an unweighted mean. One evaluation user is invalid because Claude-Opus-5 refuses the PersuadeGym role on every turn, yet that block still supplies the low end of the main gain range. An additional baseline, InfoPO, is reported only in the appendix, where it actually matches or exceeds the GPT-5.5-simulator baseline, and is omitted from the main table. For CSD, the coaching model is never identified, no generic-note or compute-matched control is reported, and the claimed gains are not uniform: CSD loses to MIMESIS-only GRPO in several user blocks under an unweighted environment mean and regresses on the held-out Intention environment under every evaluation user. More fundamentally, Suh et al. 2026 already report that a simulator fine-tuned on human utterances beats role-playing LLMs as a training environment, with human validation and cross-simulator generalization, and OSim makes the same point about assistant LLMs as simulators. This paper's controlled comparison is against GPT-5.5 role-play only, not against other learned simulators such as OSim, DITTO or HumanLM, so its incremental contribution over prior work is unclear. Finally, there is no human evaluation of either the simulator in the loop or the trained agent. The human reference in Figure 2 is unattributed, and Figure 2 includes only one non-MIMESIS trained simulator, so the claim that every trained simulator is closer to humans than every off-the-shelf model is not supported by the plotted set.

Questions for authors

Splits and decontamination. What is the relationship between the SOUL tasks used for simulator RL and the SOUL-Index evaluation? Were official train/test splits used, and were evaluation prompts or conversations ever used in mid-training or RL? Please report SOUL-Index on tasks never used in simulator RL and with a judge other than GPT-5.5, confirm that the SimulatorArena evaluation is disjoint from the SimArena tasks in Figure 9 and Table 7, and list the mid-training corpora with confirmation that WildChat, PRISM, τ-bench human turns, and SimulatorArena are excluded.

Aggregation definitions. How are the SOUL axes and the overall index computed, and how is the agent "Avg." defined? Please reconcile Table 1 with Table 7, report per-environment results, and show whether the conclusions hold under an unweighted mean and mean rank.

Variance and invalid comparisons. How many independent agent-training runs underlie the reported results, and what are the confidence intervals for the per-user deltas? How should the Claude-Opus-5 refusal block be interpreted, and is it excluded from the "all nine users" claims? Why is InfoPO absent from the main comparison?

CSD attribution. Which model produces the coaching notes, and what is the teacher in Eq. 8? Please add controls for a note built only from the public utterance, a random or generic note, and a compute-matched condition without simulator feedback, and report sensitivity to the loss weight and gate sharpness. Why does CSD regress on the held-out Intention environment?

Prior-work comparison and human validity. Given Suh et al. 2026 and OSim, how does MIMESIS as a training environment compare with other learned simulators such as OSim, DITTO and HumanLM? What are the source, sample and protocol of the human reference in Figure 2, and can the authors provide a small human-user evaluation or revise "new users" to "new simulators"?

5 comments
AuthorsOpenReview ↗

Response to Reviewer pukn (1/4)

We thank the reviewer for the detailed and constructive review. Below, we address each weakness (W) together with the corresponding questions (Q):

  • W1 (Q1, Q2) SOUL as held-out evaluation · index reconstruction · contamination of external benchmarks.
  • W2 (Q2, Q3) seeds · aggregation · Claude-Opus-5 block · InfoPO.
  • W2 (Q4) CSD attribution and non-uniform gains.
  • W2 (Q5) prior work · human validity.

W1.1 / Q1 — "The headline SOUL-Index result is not a held-out evaluation"

SOUL is evaluated on held-out instances, with a judge different from the training reward model.

  • Held out. No evaluation instance is used in training. Table 8 lists 30,531 RL prompts and 3,100 disjoint evaluation instances (App. B.1). All models are trained on fixed schedules (mid-training ≈1 epoch, ThoughtTrace SFT 3 epochs, joint RL 500 steps), and we report the final checkpoint. The evaluation instances are never used for validation or checkpoint selection, and the curves in Fig. 9 only monitor training. We conjecture that the performance on SOUL could be higher if we train our model for longer as the rewards are still increasing.
  • Like-for-like comparison. OdysSim introduced SOUL together with RL training on the same task suite, so the released OSim and Ditto simulators are trained on the same task families. The trained-vs.-trained comparison is therefore MIMESIS vs. OSim and Ditto. MIMESIS-9B leads OSim-8B by 10.1 points, Ditto-8B by 13.5 points, and the same-backbone Osim3.5-9B by 9.1 points.. Frontier models are included as reference points, yet the comparison is not strictly trained vs. zero-shot: all SOUL tasks are built from previously released public datasets, so they are not entirely out-of-distribution for recent frontier models. SOUL shares task families with our RL data, but MIMESIS is also strong on benchmarks from domains it was never trained on (e.g. τ-USI), which shows that it generalizes beyond the training domains.
  • Judge independence. Every model is re-judged by GPT-5.6-Sol, where MIMESIS-9B ranks first with both partners (Tables 9–10):
TypeModelClaude-Opus-5.5 partnerGPT-5.6-Luna partner
OursMIMESIS-9B71.671.2
MIMESIS-4B68.868.8
FrontierClaude-Opus-570.570.3
GPT-5.569.969.6
Gemini-3.8-Flash66.966.1
Trained simulatorsOsim3.5-9B (same backbone)62.562.2
OSim-8B61.561.4
OSim-4B58.558.5
Ditto-8B58.157.4
HumanLM52.452.2
Sotopia-7B39.839.5
PretrainedQwen3.5-9B53.553.9
Qwen3-8B51.851.4
Qwen3.5-4B48.147.5

Finally, on the benchmarks the reviewer asked about, whose tasks are never used in mid-training or simulator RL, MIMESIS reaches 94.0 on RealUserSim (compared with 80.6 for Claude-Opus-5), the lowest calibration error on τ-USI (ECE 0.069, compared with 0.135 for OSim-8B), the highest mean on PRISM (80.1), and the lowest Turing distance on SimulatorArena (38.7).

AuthorsOpenReview ↗

Response to Reviewer pukn (2/4)

W1.2 / Q2 — "The index itself is not verifiable"

The revision defines the index explicitly, and it is exactly reproducible (App. B.1). The five axes contain 4, 1, 4, 9 and 5 benchmarks, and each axis score is the mean of its benchmarks. The Overall is the mean of all 23 benchmark scores, so it can be reconstructed from the axis scores:

SOUL=123∑b=123sb=4⋅CONV+SS+4⋅COG+9⋅ROLE+5⋅EVAL23.\mathrm{SOUL}=\frac{1}{23}\sum_{b=1}^{23} s_b=\frac{4\cdot\mathrm{CONV}+\mathrm{SS}+4\cdot\mathrm{COG}+9\cdot\mathrm{ROLE}+5\cdot\mathrm{EVAL}}{23}.

In other words, the Overall is a mean over benchmarks rather than an unweighted mean of the five axis scores.

W1.3 / Q1 — Contamination of RealUserSim, PRISM, τ-USI and SimulatorArena

No example from our evaluation sets appears in our training data. Specifically, neither mid-training nor RL uses the evaluation conversations of RealUserSim PT3, PRISM, τ-USI or SimulatorArena, nor any of the SOUL held-out instances. This exclusion is applied to individual conversations and prompts. For transparency, the updated PDF also lists the 62 mid-training corpora in App. A.1.

W2.1 / Q3 — "Single-run … no seed count, confidence interval, or statistical test"

All main agent results now use five independent training seeds, with 95% CIs and paired tests. Fig. 5 and the alternative-simulator figure (Fig. 16, formerly Fig. 13) now show all five seeds with 95% CI bands. Paired over the nine users, the simulator effect is +3.52 (95% CI [+2.67, +4.36]) and the CSD effect is +2.06 ([+1.40, +2.73]), both with p < 0.001. The effects also hold at the per-user level: all 18 per-user Welch intervals lie above zero, and the one closest to zero, the CSD effect under Gemini, is +0.71 [+0.01, +1.41]. Finally, CSD separates from both baselines at around step 35, and the intervals stay disjoint thereafter (Fig. 5).

At the final step (100), the five-seed means are:

MethodValidation reward [95% CI]Training reward
InfoPO0.95 [0.92, 0.98]1.72
GRPO0.99 [0.91, 1.08]1.93
CSD1.31 [1.25, 1.36]2.56

W2.2 / Q2 — "The 'Avg.' column … undocumented weighting"; unweighted mean and mean rank

Avg. follows the official UserRL/InfoPO evaluation code, which weights every evaluation instance equally [1,2]:

Avg=∑gngSg∑gng,\mathrm{Avg}=\frac{\sum_g n_g S_g}{\sum_g n_g},

where SgS_g is the mean score of gym gg and ngn_g its number of instances. The paper now states this definition. Per-environment results are in Tables 17–18.

For convenience, the table below averages the per-environment entries of Tables 17–18 over the nine evaluation users († marks gyms held out from agent training; MR, the mean within-environment rank, ranks all five methods, including the Qwen3-8B base agent before RL; bold marks the best condition).

GymGRPO + GPT-5.5InfoPO + MIMESIS-9BGRPO + MIMESIS-9BCSD + MIMESIS-9B
Travel13.1913.0413.6317.49
Turtle26.8326.8826.8631.32
Function5.747.6919.7717.16
Tau14.5315.2715.5715.26
Persuade34.6440.3744.0145.49
Intention†165.09176.25195.83190.44
Telepathy†56.6456.2957.0662.84
Search†45.9944.0149.4751.27
Avg. (instance-weighted)26.2126.8429.7331.79
MR ↓3.453.172.221.66

The original submission already addressed differences in per-environment magnitude by also reporting the mean within-environment rank. MR is a standard multi-task-learning metric for summarizing performance across tasks with different scales (e.g., Nash-MTL, FAMO). Because the scores of the gyms differ in magnitude, we regard MR as the primary summary of agent-training performance across gyms, which is why it closes every per-environment table. Importantly, MR gave the same ordering as Avg. in the submission (GRPO + GPT-5.5 3.42, GRPO + MIMESIS-9B 2.15, CSD 1.75), and it gives the same ordering again in the revision with five seeds (3.45, 2.22, 1.66).

As requested, we additionally report aggregations that weight environments equally. In the table below, each cell counts the users (out of 9) for which the difference is positive (mean difference in parentheses; the first row is the official Avg., for reference).

AggregationSimulator effectCSD effect
Instance-weighted Avg. (official)9/9 (+3.52)9/9 (+2.06)
Mean within-environment rank9/98/9
Equal weight, scale-normalized per environment9/98/9
Held-out gyms only9/9 (+8.16)7/9 (+1.20)

The orderings hold across these aggregations. This scale effect also accounts for most of the user blocks in which CSD appeared to lose under the unweighted mean in the single run.

[1] UserRL: Training Proactive User-Centric Agent via Reinforcement Learning

[2] InfoPO: Information-Driven Policy Optimization for User-Centric Agents

AuthorsOpenReview ↗

Response to Reviewer pukn (3/4)

W2.3 / Q3 — The Claude-Opus-5 block*

We thank the reviewer for catching this, and we have re-run the block with measured scores. We note that the issue was a property of the evaluation user rather than a weakness of our method. Claude-Opus-5 refused the user role in one gym (PersuadeGym) while behaving normally in all the others. Under the initial prompt, Claude-Opus-5 returned an empty refusal (stop_reason = "refusal") on all 672 PersuadeGym turns while Claude-Sonnet-5 answered the identical input normally; PersuadeGym then substituted the fixed utterance "I'm having trouble understanding your argument right now.", left the stance unchanged, and assigned zero reward. The resulting zero-reward fallbacks affected every training condition equally, so no comparison was biased.

To obtain valid scores, we re-ran the block with a revised role prompt, so every "all nine users" claim now rests on measured scores. In addition, the conclusions hold when Claude-Opus-5 is excluded entirely (simulator effect +3.80, CSD effect +1.91).

W2.4 / Q3 — InfoPO

InfoPO is now included in the main table (Table 2). We initially kept it in the appendix for space and readability. The main table focused on the controlled comparison between GRPO (the UserRL objective, which we label by its RL algorithm, GRPO, for clarity) and CSD, and we wanted to avoid confusing readers with additional methods, setups, and paper names. With the additional space in the revision, we add InfoPO to Table 2 and describe its setup. When trained with the same MIMESIS-9B simulator, InfoPO scores 26.84, below both GRPO + MIMESIS-9B (29.73) and CSD + MIMESIS-9B (31.79) under all nine users.

W2.5 / Q4 — CSD: coaching model, controls, sensitivity

Both the coach and the teacher are built from the agent model itself. The coach is the agent model run without gradients. Given the public history, the sampled response, and the simulator's private thought and next utterance, it writes a note sampled at temperature 0.7 with at most 300 tokens. The teacher is a stop-gradient copy of the current policy that additionally conditions on this note and re-scores the already-sampled tokens. Importantly, neither the rollout policy nor the deployed agent ever sees a note.

Formally, let GG be the coach, HtH_t the public history, yty_t the sampled response with tokens yt,ky_{t,k}, ztz_t and xt+1x_{t+1} the simulator's private thought and next utterance, θˉ\bar\theta a stop-gradient copy of θ\theta, and C\mathcal C the agent-token positions of coached responses:

ht=G(Ht,yt,zt,xt+1),Δt,k=log⁡πθˉ(yt,k∣Ht,ht,yt,<k)−log⁡πθ(yt,k∣Ht,yt,<k),h_t=G(H_t,y_t,z_t,x_{t+1}),\qquad \Delta_{t,k}=\log\pi_{\bar\theta}(y_{t,k}\mid H_t,h_t,y_{t,<k})-\log\pi_\theta(y_{t,k}\mid H_t,y_{t,<k}),

gt,k=σ(βsg(Δt,k)),LCSD=−E(i,t,k)∼C[gi,t,klog⁡πθ(yi,t,k∣Hi,t,yi,t,<k)],g_{t,k}=\sigma\big(\beta\mathrm{sg}(\Delta_{t,k})\big),\qquad \mathcal L_{\mathrm{CSD}}=-\mathbb E_{(i,t,k)\sim\mathcal C}\big[g_{i,t,k}\log\pi_\theta(y_{i,t,k}\mid H_{i,t},y_{i,t,<k})\big],

where σ\sigma is the sigmoid and sg\mathrm{sg} the stop-gradient. The agent minimizes Lagent=LGRPO+αLCSD\mathcal L_{\mathrm{agent}}=\mathcal L_{\mathrm{GRPO}}+\alpha\mathcal L_{\mathrm{CSD}} with α=0.01\alpha=0.01 and β=5.0\beta=5.0. CSD is thus a weighted-likelihood loss on the agent's own sampled tokens: tokens that the coached teacher finds more probable than the uncoached student receive larger weights.

The controls requested by the reviewer isolate relevant simulator feedback as the source of the gain. All coaching conditions use the same number of coach calls, teacher passes, selected tokens, rollouts, and training steps, so they differ only in the content of the note; the noise control compares independent GRPO seeds.

Coach inputΔ Avg. vs. GRPO + MIMESIS-9B (mean over 9 users)
Independent GRPO seeds (noise control)−0.04
Generic note+0.04
Note from an unrelated trajectory−0.04
Public history only+0.68
Public history + next user utterance+1.16
Public history + private thought+1.59
Full note+2.06

Generic and mismatched notes are statistically equivalent to no coaching (p_equiv = 0.037 and 0.007). Thus, neither extra computation nor the form of a note explains the gain. In contrast, simulator feedback adds +1.39 over public-only notes (p < 0.001), and both feedback channels contribute.

Theoretical support (App. E). For small step sizes, a GRPO + CSD step provably improves on the GRPO step whenever the coaching weights are sufficiently well calibrated to advantages.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2)

We thank all reviewers for their constructive comments, which have helped us strengthen our experimental evaluation and better highlight the contributions of MIMESIS. We are encouraged that the reviewers recognized the importance of realistic user simulation (wcop) and of generalization beyond the simulators used during agent training (gYmk). Reviewer pukn found our simulator clearly stronger than off-the-shelf baselines, with consistent gains from reasoning supervision and realistic-behavior training. Reviewers also viewed the controlled simulator-swap experiment, in which the agent improves under all nine evaluation users, as the right test of our central claim. Reviewer wcop acknowledged the breadth of the empirical evaluation and the organization of the paper, and Reviewer gYmk pointed to our combination of user-interaction mid-training and reasoning-oriented post-training as a strength.

Stronger simulator comparisons and clearer evaluation. We added Osim3.5-9B - an Osim-recipe baseline on the same Qwen3.5-9B backbone and evaluated SOUL with a judge distinct from the training reward model. MIMESIS retains a 9.1-point advantage over this matched-backbone baseline with the Claude-Opus-5.5 interaction partner and a 9.0-point advantage with GPT-5.6-Luna.

Table R1. Simulator quality and task-outcome calibration. SOUL uses the GPT-5.6-Sol judge with the Claude-Opus-5.5 partner. Human gaps are computed from success rates.

ModelSOUL ↑RealUserSim ↑τ-USI ↑SimulatorArena ↓τ-bench success (%)Human gap (pp) ↓
MIMESIS-9B71.6 ± 0.194.0 ± 0.380.17 ± 0.2138.7 ± 0.766.1 ± 2.42.4
MIMESIS-4B68.8 ± 0.189.7 ± 0.178.03 ± 1.0742.0 ± 1.565.5 ± 2.51.8
Claude-Opus-570.5 ± 0.080.6 ± 0.269.34 ± 0.5742.3 ± 1.984.4 ± 0.520.8
GPT-5.569.9 ± 0.176.5 ± 0.676.69 ± 1.8348.7 ± 0.382.4 ± 0.018.8
Gemini-3.8-Flash66.9 ± 0.167.2 ± 0.373.13 ± 0.9246.7 ± 0.782.8 ± 1.019.2
Osim3.5-9B62.5 ± 0.262.3 ± 0.165.3 ± 1.748.1 ± 0.449.7 ± 1.213.9
OSim-8B61.5 ± 0.557.7 ± 0.380.44 ± 1.1648.7 ± 0.353.3 ± 2.110.3
Qwen3.5-9B53.5 ± 0.739.4 ± 0.657.82 ± 1.5246.3 ± 2.148.7 ± 2.114.9
Human reference——91.75—63.6 ± 2.40.0
AuthorsOpenReview ↗

General response (2/2)

Addressing shared concerns about CSD. Several reviewers asked about the effectiveness and robustness of CSD. In response, we have substantially revised the manuscript with additional experiments and a new theoretical analysis. We repeated the main agent-training comparisons with five independent random seeds, expanded the coaching ablations and hyperparameter sensitivity studies, and evaluated CSD with additional training simulators. We also added a theoretical analysis (App. E) that characterizes the policy preferences induced by CSD and derives sufficient conditions under which a CSD-augmented update improves expected return over the corresponding GRPO update.

Table R2. Agent generalization across nine unseen evaluation users. Mean over five training seeds; ± is a 95% CI across seeds.

Evaluation userGRPO with GPT-5.5InfoPO with MIMESIS-9BGRPO with MIMESIS-9BCSD with MIMESIS-9B
Ditto-8B27.39 ± 0.8526.73 ± 0.1829.83 ± 0.4832.44 ± 0.71
OSim-8B25.57 ± 0.4925.78 ± 0.5829.14 ± 0.5230.08 ± 0.50
HumanLM-8B27.03 ± 0.3926.65 ± 0.7630.73 ± 0.6433.01 ± 0.85
Sotopia-7B24.53 ± 0.4025.99 ± 0.3728.84 ± 0.7631.70 ± 0.40
GPT-5.626.51 ± 0.7227.78 ± 0.9531.09 ± 0.3533.06 ± 1.00
Claude-Opus-527.77 ± 0.8828.02 ± 0.5628.99 ± 0.5332.27 ± 0.37
Claude-Sonnet-526.32 ± 0.5827.88 ± 0.9129.92 ± 0.6531.43 ± 0.27
Gemini-3.8-Flash26.29 ± 0.5627.78 ± 0.2930.99 ± 0.6531.70 ± 0.54
Kimi-K324.50 ± 0.7324.94 ± 0.1528.04 ± 0.9330.43 ± 0.59
Mean26.21 ± 0.5026.84 ± 0.4329.73 ± 0.4731.79 ± 0.45

Training with MIMESIS-9B instead of GPT-5.5 improves the agent under all nine users (+3.52, 95% CI [+2.67, +4.36]), and CSD adds a further +2.06 ([+1.40, +2.73]), again under all nine; both effects are significant (p < 0.001, paired over users).

Additional controls and clearer scope. We specified the coach and teacher models, added generic-note, mismatched-note, and feedback-channel controls that use the same number of coaching operations, and tested sensitivity to the auxiliary loss weight and the gate sharpness. We also trained agents with alternative simulators, expanded the discussion of related simulator methods, and clarified the training stages and benchmark definitions. The main new or expanded analyses can be found in:

  • App. B.1, Tables 8–10: SOUL splits, aggregation, evaluation configurations, and the same-backbone comparison.
  • App. D.3, Tables 16–18: five-seed agent-training results, uncertainty estimates, and detailed cross-simulator evaluation.
  • App. D.4, Fig. 16: agent training with alternative simulators.
  • App. D.5, Figs. 17–18: coaching-information controls and hyperparameter sensitivity.
  • App. E: theoretical analysis of CSD.
  • App. F.3 and App. G: the corrected Claude-Opus-5 evaluation and the limitations of the generalization claims.

These revisions strengthen the evidence for MIMESIS as a user simulator and agent-training environment, clarify the role of CSD, and make the evaluation more transparent. We address each reviewer's specific concerns below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.