acceptodds
Under review as a conference paper at ICLR 2027

Training User Simulators with Latent Intrinsic Feedback

Abstract

User simulators are gaining attention as environments to train LLM assistants that can better cope with the nuances of multi-turn conversations with real users. Prior work on user simulation focuses primarily on generating realistic user utterances, leaving unexplored their potential as a source of direct feedback on the assistant's responses. Such an evaluation signal may not always be explicitly communicated in language, but can instead reside in the latent reasoning of users. In this work, we introduce user simulators with intrinsic feedback, which are models equipped with latent evaluation states: internal thoughts, reactions, and scalar ratings that evaluate the assistant from a first-person user perspective at each conversational turn. We train LLM-based simulators using interaction logs between real users and assistants and demonstrate that the use of intrinsic feedback improves simulation realism and behavioral alignment with real human users. Further, we show that evaluation signals generated by our trained simulators are better correlated with human judgments, compared to evaluations generated by standard prompted simulator baselines. Lastly, our experiments reveal that simulators with intrinsic feedback provide better reward signals for multi-turn RL training, producing more capable LLM assistants while simultaneously improving computational efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
User Simulation, User Models, Multi-turn RL Training, LLM Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_gBvDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes user simulators with latent evaluation states that generate both user responses and turn-level intrinsic feedback on the assistant. The simulator is trained on WildChat interactions with synthetic LES annotations and is further used to provide reward signals for multi-turn RL, reducing the need for an external judge model.

Critical strengths
  1. The idea of user simulators with intrinsic feedback is interesting and practically useful, allowing the simulator to provide turn-level evaluations directly during multi-turn RL.
  2. The experiments are relatively comprehensive, covering multiple model scales and families, feedback behavior, simulation realism, human rating correlation, ablations, and multi-turn RL.
Critical weaknesses
  1. The LES supervision is generated by Gemini3.1-Pro. The paper does not report human verification or agreement analysis for these synthetic labels, leaving their quality unclear.
  2. Most annotated turns are Neutral, yet Internal Thought and Rating are still generated for these turns. Since these latent evaluations are not directly observed from users, it remains unclear how reliably they reflect the actual user state.
  3. The benefits of full LES training are not consistent across evaluations. Standard SFT achieves higher Critique F1 across all Gemma scales, the Rating-only ablation achieves higher F1 than full LES, and the gains in human rating correlation are not consistently significant across model scales and subsets.
Questions for authors
  1. Could the authors provide human validation or agreement analysis for the generated LES labels, especially Internal Thought and Rating for Neutral turns?
  2. Related work such as ThoughtTrace collects direct self-reported thoughts and finds that latent user states are difficult for frontier LLMs to infer from conversation context. Could the authors further validate whether the Gemini-generated LESs reflect direct user evidence?
2 comments
AuthorsOpenReview ↗

Response to Reviewer gBvD

Thank you so much for your efforts reviewing our paper, we really appreciate your time and valuable suggestions! We respond to your concerns and questions below.

(1) Human Verification of LES labels: We conducted a human annotation of the LES for 100 randomly sampled conversations, with 20 conversations sampled from each rating level. The three human annotators achieved a good average Pearson correlation of 0.74 with the Gemini-generated labels. We hope this provides further quality guarantees about our approach for synthetic labeling

(2) Clarification on Neutral turns: We considered turns that are not explicit praises or critiques as “Neutral”, but consist of other reactions such as follow-ups, re-prompting, or topic shifting, for which the ratings can vary. We agree that the term Neutral may not be the best terminology here and suggest revising it to “Other” reactions for less confusion.

(3) Clarification on full LES benefits: We clarify that the training ablations in Table 2 were conducted to better understand the impact of the different LES components. We find that full LES training is primarily beneficial for improving simulation realism compared to SFT models, which is a crucial aspect of simulator quality. However, this comes at the cost of a lower critique F1 compared to training with ratings alone, which achieves the strongest critiquing ability and even outperforms the SFT model. Given this tradeoff, we see the main benefit of full LES training as combining strong evaluative capabilities, comparable to those of prompted simulators, with substantially better simulation realism. We have revised our discussion of these ablation results in Section 4.3 to better reflect this finding.

(4) Comparison to ThoughtTrace: We clarify that in ThoughtTrace, the self-reported thoughts of the human users were completely open-ended, which indeed makes it difficult for LLMs to infer. In contrast, the thoughts in our LES are scoped to evaluate whether the assistant followed the user’s instructions, based on the conversation history and what the next user turn was, which is relatively easier for LLMs to infer. The strong agreement levels of our human verification also supports this.

Thank you so much again for your review! We hope our additional human verification and clarifications addressed your concerns, and we would really appreciate it if you would increase your score.

Dear Reviewer gBvD,

We wanted to remind you that the discussion period ends tomorrow. We would appreciate it if you could take a moment to check our response and let us know if you have any additional questions or clarification requests.

Thank you for your time!

Reviewer_sCHUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores the potential of user simulators to provide intrinsic feedback during multi-turn interactions with LLM assistants. The authors propose user simulators equipped with Latent Evaluation States (LESs), which generate internal thoughts, reactions, and ratings at each conversational turn to better capture realistic user behavior. Experimental results demonstrate that the proposed approach achieves improvements over several baseline models in terms of simulation realism and feedback quality. Additional experiments further demonstrate the effectiveness and computational efficiency of using such simulators to train LLM assistants through reinforcement learning.

Critical strengths
  1. The authors explore an interesting direction for user simulator modeling by introducing a chain-of-thought-like mechanism that explicitly models multiple aspects of user feedback, including ratings, internal thoughts, instruction following, reactions, and feedback summaries. This design provides a more comprehensive representation of user behavior and has the potential to improve simulation realism.

  2. The paper is generally well-organized and easy to follow. However, I believe that the Background and Related Work sections could be better integrated to improve the overall coherence and reduce redundancy.

  3. One particularly valuable contribution is the demonstration that user simulators can serve not only as interactive counterparts in multi-turn conversations but also as providers of reward signals for reinforcement learning. The experimental results suggest that this approach can improve assistant performance while substantially reducing computational costs by eliminating the need for a separate reward evaluation model during RL training.

Critical weaknesses
  1. Insufficient comparisons with existing user simulation methods. The experimental comparisons are primarily conducted among different configurations of the authors' own framework, including prompted models, standard supervised fine-tuning, and the proposed LES-based training. However, the paper does not provide direct empirical comparisons with other representative user simulation approaches, several of which are already discussed in the Related Work section. Such comparisons would help better establish the effectiveness and novelty of the proposed method relative to existing research.

  2. Unclear relationship between the distilled latent evaluations and generated user responses. The proposed framework uses teacher-generated latent evaluation states to supervise the training of user simulators. However, it remains unclear how strongly these evaluations are related to the actual user responses generated by the simulator. In particular, does the model learn a consistent relationship between its internal evaluations and subsequent user behavior, or does it simply learn to generate both types of outputs through supervised fine-tuning? Further analysis would help clarify whether the latent evaluation states genuinely contribute to realistic user simulation.

  3. Potential coupling and bias between the user simulator and the RL-trained assistant. The computational efficiency of the proposed RL framework mainly comes from the fact that the user simulator simultaneously acts as a reward provider, removing the need for a separate reward model. However, this raises an important concern: could the resulting assistant become overly optimized toward the specific feedback preferences or behavioral patterns of the user simulator? Such coupling may introduce systematic biases and limit generalization to real users or other simulation environments. The current experiments do not sufficiently investigate this potential issue.

Questions for authors

Regarding Tables 2 and 3: The ablation results are particularly interesting. Could the authors provide a more detailed explanation of the observed trends? The effects of different latent evaluation state components and user-intent conditioning appear to vary across evaluation metrics and model sizes. In particular, the improvements do not seem entirely consistent. What factors might explain these variations, and what do they suggest about the contribution of each component?

Regarding multi-turn interaction behavior: Since the proposed method aims to construct realistic user simulators, an important application is to engage in multi-turn conversations with LLM assistants. Although the paper evaluates user simulation quality and uses multi-turn rollouts for RL training, it provides limited qualitative analysis of the actual interaction process. Could the authors provide representative multi-turn conversation examples comparing the proposed LES-based simulators with the baselines? Such examples would help illustrate how the generated latent evaluations influence subsequent user responses and whether they lead to more realistic interaction patterns.

4 comments
AuthorsOpenReview ↗

Response to Reviewer sCHU

Thank you so much for your efforts reviewing our paper, we really appreciate your time and valuable suggestions! We respond to your concerns and questions below.

(1) Comparisons to additional simulator baselines: We have run additional user simulation models from past work (HumanLM-8B and OSim-4B) on the SimulatorArena and report the results below. Our trained LES models still achieve the highest correlations with real human evaluations. We have added these additional comparisons to our Appendix C1.

SimulatorDocMathAll
Gemma3-4b-IT0.0730.3780.244
Gemma3-4b-LES0.2730.4400.384
OSim-4B0.2850.2330.296
OSim-8B0.2540.3080.285
HumanLM-8B0.2160.1650.215
Gemma3-12B-IT0.1580.4020.295
Gemma3-12B-LES0.3300.4120.401

(2) Consistency between latent evaluations and user response: We agree that this is an interesting evaluation. We indeed expect the simulator’s response to be consistent with how its latent thoughts evaluated the assistant. To further verify this, we conducted a human evaluation of 50 randomly sampled outputs generated by our Gemma3-4B-LES on a held-out set from WildChat. The two human annotators labeled 98% of the simulated responses to be fully consistent with the internal reasoning trace and evaluation.

(3) Bias towards the simulator in RL: This is a good point, thanks for the observation. We note that all of our RL results are evaluated on WildBenchV2 with an independent judge, which is completely independent from the simulator used in training. This was done specifically to measure if the gains carry over beyond the simulator’s own preferences. To further verify that the trained Qwen assistants do not overly optimize for a specific simulator, we performed cross-simulator testing of the assistant trained under each setup with Gemma3-4B-LES and Gemma3-4B-IT simulators, as well as external simulators based on prompting Llama3.2-3b-Instruct and OSim-8B. The table below shows the average reward achieved by each trained assistant on a heldout set of 500 task intents from Wildchat when conversing with each simulator. All RL-trained assistants improve over the initial model checkpoint when conversing with every simulator, including ones they never interacted with. We hope this addresses your concern about the assistant overfitting to the feedback preferences of the simulator used as its reward provider. We have also added these additional results to our Appendix C3.

Assistant Training SetupGemma3-4b-LESGemma3-4b-ITLlama3.2-3BOSim-8b
base (no RL)3.8753.8043.4553.847
RL w/ Gemma3-4b-LES (Intrinsic)4.3043.9604.0494.276
RL w/ Gemma3-4b-IT (Intrinsic)4.3023.9714.0444.258
RL w/ Gemma3-4b-LES + Judge4.2543.9673.9734.266
RL w/ Gemma3-4b-IT + Judge4.2103.9644.1184.247

(Q1) Clarification on the LES training ablation trends: We clarify that the training ablations in Table 2 were conducted to better understand the impact of the different LES components. We find that full LES training is primarily beneficial for improving simulation realism compared to SFT models, which is a crucial aspect of simulator quality. However, this comes at the cost of a lower critique F1 compared to training with ratings alone, which achieves the strongest critiquing ability and even outperforms the SFT model. Given this tradeoff, we see the main benefit of full LES training as combining strong evaluative capabilities, comparable to those of prompted simulators, with substantially better simulation realism. We have revised our discussion of these ablation results in Section 4.3 to better reflect this finding!

(Q2) Qualitative Study Examples: We ran a Turing-style human study to measure realism on 150 samples SimulatorArena. Two annotators saw two candidate next user messages, one written by the real user and one generated by a simulator, and picked the one they thought was real. The fool rate for each simulator is summarized in the Table below. The annotators mistook LES messages for the real user’s 59% of the time, so they were indistinguishable from real users The SFT simulator reached 48%, and prompted Gemma3-4B-IT reached only 19%. The results of this human study are consistent with our statistical metrics and perplexity evaluations. We have also added several example outputs from the LES simulators at the end of our Appendix.

SimulatorFool Rate
Gemma3-4b-LES59.0%
Gemma3-4b-SFT48.0%
Gemma3-4b-IT19.0%

Thank you so much again for your review! We hope our additional results and clarifications addressed your concerns, and we would really appreciate it if you would increase your score.

Dear Reviewer sCHU,

We wanted to remind you that the discussion period ends tomorrow. We would appreciate it if you could take a moment to check our response and let us know if you have any additional questions or clarification requests.

Thank you for your time!

Thank you for your response. I have carefully read the rebuttal, and it could strengthen the paper if these new results were included in the final version. I think my score is good enough. Thank you for your effort.

Reviewer_5hXkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

When LLMs serve as user simulators, they usually only give the response but rely on another LLM judge to provide feedback. This paper proposes latent evaluation states (LESs), which let the simulator emit an internal thought, including a rating, feedback, etc., before generating responses. Compared with the SFT baseline method, LES shows better performance in Simulation Realism, but worse in Feedback Behavior.

Critical strengths
  1. The problem setting is realistic and interesting.
  2. The experiment is clear and complete.
  3. Easy to reproduce, as the prompts are clearly stated.
Critical weaknesses
  1. Circular use of Gemini3.1-Pro. The training labels of LES are generated by Gemini3.1-Pro. But as stated in Line 407, it's also used as the task-specific checklist judge. So the advantage in Figure 5 may reflect a more aligned evaluator preference rather than user modeling.
  2. It's not clear if the benefit comes from intrinsic generation of the rating or from distillation from Gemini. A fairer baseline should be a Gemma3-4B evaluator fine-tuned on the same (context, rating) pairs as LES. It only evaluates but does not simulate a response.
  3. Some uncited related works. [1][2]
  4. The Critique F1 does not support the "improves critiquing behaviors" statement. In Table 1, LES shows lower F1 Scores than SFT. In Table2, Rating-only shows the highest Feedback Behavior.
  5. Concern about the effectiveness on larger scales. In Table 4, the advantage of LES over IT shrinks at the 27b scale. In the Math domain, LES even performs worse. Besides, the SimulatorArena's own simulator baselines, Gemini3.1-Pro, and an SFT simulator are missing.

[1] Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models. [arxiv 2025] [2] RLVER: Reinforcement Learning with Verifiable Emotion Rewards for Empathetic Agents. [ICLR 2026]

Questions for authors
  1. The performance on Feedback Behavior and Simulation Realism shows a reversed trend, like in Table 2. Can you explain the reason? Is it for all simulators or specifically for LES?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 5hXk

Thank you so much for your efforts reviewing our paper, we really appreciate your time and valuable suggestions! We respond to your concerns and questions below.

(1) Using another judge LLM for evaluation: We re-ran the evaluation for the results on WildBenchV2 of Table 5 using a different judge LLM (Claude Haiku-5.5) instead of Gemini3.1-Pro and report the results below. We find similar trends in the results, where LES models still achieve the highest scores, which addresses doubts of a bias in the judge LLM. We also note that this evaluation is based on an explicit checklist rubric provided by the benchmark, which avoids judge biases.

SimulatorGemini3.1-ProClaude Haiku 5.5
—28.7825.39
Gemma3-4b-IT (Judge)39.7935.64
Gemma3-4b-IT (Intrinsic)40.5235.35
Gemma3-4b-SFT (Judge)40.5935.61
Gemma3-4b-LES (Judge)41.0836.10
Gemma3-4b-LES (Intrinsic)41.4436.40

(2) Clarification on rating generation: We would like to clarify that our goal is not to build a better standalone evaluator, as in an LLM judge, but a user simulator that produces its own feedback which importantly eliminates the need for an external reward model. We trained on Gemini-annotated ratings that evaluate instruction following capabilities of the assistant as part of the method. We note that these ratings correlate well with humans, which we verified on a small random sample of 100 conversations, where annotators achieved a strong correlation of 0.74 with Gemini-generated labels.

(3) Additional references: Thanks for sharing these additional references, we have incorporated them into our related work.

(4) Modifying paragraph header: In our discussion of the results in the first paragraph of Section 4.2, we mention how feedback behavior is improved by LES models compared to prompted simulators, yet the F1 score is slightly worse compared to SFT models. We have updated the paragraph header to remove the general claim and better reflect our discussion. Thanks for pointing this out.

(5) Math results on SimulatorArena: Your observation that the improvements by LES simulators on the math domain of SimulatorArena is not as strong as the document writing domain where we achieve statistically significant results is correct. We believe this is because the WildChat data used to train the simulators consist more of document writing tasks with less math focused tasks, making them out-of-distribution compared to the simulator’s training data. We hope that with the release of better and more domain-diverse human-LLM interactions logs in the future, better results can be achieved. We have added this to our discussion of the results.

(Q1) Clarification on the LES training ablation trends: We clarify that the training ablations in Table 2 were conducted to better understand the impact of the different LES components. We find that full LES training is primarily beneficial for improving simulation realism compared to SFT models, which is a crucial aspect of simulator quality. However, this comes at the cost of a lower critique F1 compared to training with ratings alone, which achieves the strongest critiquing ability and even outperforms the SFT model. Given this tradeoff, we see the main benefit of full LES training as combining strong evaluative capabilities, comparable to those of prompted simulators, with substantially better simulation realism. We have revised our discussion of these ablation results in Section 4.3 to better reflect this finding.

Thank you so much again for your review! We hope our additional results and clarifications addressed your concerns, and we would really appreciate it if you would increase your score.

Thank you for the clarifications. Here are some follow-up concerns.

To (1): The Claude Haiku 5.5-evaluated results are too close among the 5 types (range within 1.05). Besides, the satisfaction of Gemma3-4b-IT (Intrinsic) is worse than Gemma3-4b-IT (Judge). These observations challenge the effectiveness of Intrinsic feedback.

To (2): I understand your target is a user simulator that produces its own feedback and eliminates the need for an external reward model. However, your response fails to answer whether the better performance comes from intrinsic generation of the rating or simply from distillation from Gemini. I ask this question because it's trained on Gemini's ratings, while the external judge model (Qwen2.5-14B-Instruct) has not.

To (5): Part of my concern remains unanswered. The response stays in explanation without supporting evidence.

AuthorsOpenReview ↗

Response to Reviewer 5hXk

Thank you for clarifying your concerns!

(1) Value of Intrinsic Feedback: We agree that the performance improvement of LES over the prompted simulators in RL training are minimal, as we have noted in our discussion of the results in Section 5.2. We clarify that the main benefit of intrinsic feedback in RL is cutting the computational cost by more than half compared to using an external judge as a reward model, which holds for both types of simulators (LES or prompted). We have further edited our writing of the results for better clarity and changed our paragraph header to mention “Intrinsic rewards from simulators are competitive with external judges”.

(2) Comparison to a rating classifier: Thank you for clarifying this point! We have trained the suggested Gemma3-4B that produces a rating given the context but does not simulate a response, using the same (context,rating) pairs from WildChat. We compare its performance on SimulatorArena compared to Gemma3-4B-LES below. The Gemma3-4B-LES model which also simulates a user response achieves higher correlations with human ratings, compared to the distilled rating prediction model, which further verifies the value of intrinsic generation of feedback from the simulator.

SimulatorDocMathAll
Gemma3-4b (rating)0.1980.2790.265
Gemma3-4b-LES0.2730.4400.384

(5) Additional SimulatorArena results: We have added the results for Gemini3.1-Pro as a prompted simulator on SimulatorArena. We clarify that the Gemma3-SFT models cannot be added to the results here because they are trained to only simulate a response but do not produce ratings. However, we have also added several past baseline simulator models (HumanLM-8B, OSim-4B, and OSim-8B). Our Gemma3-LES simulators achieve better results compared to all baselines. The Gemini3.1-Pro achieves the best correlations, which we believe is due to it being a much larger model compared to the rest of the simulators we train. Concerning the results on the math tutoring domain of SimulatorArena for LES models, we clarify that only 1.5% of WildChat consists of math-focused prompts, which makes it difficult for the trained simulators to perform better than prompted simulators on this particular domain.

SimulatorDocMathAll
Gemma3-4b-IT0.0730.3780.244
Gemma3-4b-LES0.2730.4400.384
OSim-4B0.2850.2330.296
OSim-8B0.2540.3080.285
HumanLM-8B0.2160.1650.215
Gemma3-12B-IT0.1580.4020.295
Gemma3-12B-LES0.3300.4120.401
Gemma3-27B-IT0.2250.4210.343
Gemma3-27B-LES0.3420.3900.383
Gemini3.1-Pro0.3620.4230.384

Thank you again for your feedback, we hope these additional results and clarifications have addressed your remaining concerns.

Reviewer_vCZXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper noticed the limitation of using RL to train user simulators that existing works always reply on external LLM-as-a-judge to infer reward, which may poorly capture first-person implicit user perspective and have heavy compute overhead. Motived by this, the paper proposed the Latent Evaluation States (LESs) and use them as intrinsic feedback to train user simulators. They train LES-based simulator with Gemma3 which shows promising simulation results compared to prompted baselines. They also train AI assistant with their LES-based simulator using multi-step RL, which matches setups with external judges but with less compute.

Critical strengths

This paper repurposes the latent state zt+1z _{t+1} into a structured first-person evaluation + scalar 1–5 reward, merging the environment model and the reward model into a single forward pass during open-domain LLM GRPO to cut rollout/judging compute in half. The proposed solution is well-motivated and practical.

Table 2 shows a clean empirical trade-off ablations and analysis of latent sub-states. Adding verbal thoughts and short feedback summaries bridges the gap between a discrete score and natural human phrasing, which can serve as an actionable guidance for future work designing latent states in simulators.

Critical weaknesses

My main concern is about the RL experimental gaps and if the evaluation is sound enough to prove their claims.

  1. In RL (Table 5), Gemma3-4B-LES barely beats a zero-shot prompted simulator (within noise). GRPO w/ Gemma3-4B-IT + Intrinsic (zero-shot prompted, NO simulator training) is already improved the base LLM by 11.74, whereas the full setup (GRPO w/ Gemma3-4B-LES + Intrinsic) only achieves 12.66, and the former setup has fewer FLOPs.

  2. There is no single-turn GRPO baseline (training on turn 1 alone without a user simulator), so we cannot tell how much of the 28.78% to 41.44% jump actually comes from multi-turn user simulation versus simply doing RL on 100k WildChat first-turn prompts. And there is no real-data baseline (e.g., SFT or offline RL on real human-assistant trajectories from WildChat).

  3. Their RL evaluation is at the static and surface level. They only evaluate only on static prefixes, and there is no interactive multi-turn rollouts or user study to check whether Gemma3-4B-LES drifts over long self-generated dialogues, or whether the RL-trained assistant actually collaborates better across live multi-turn interactions. Their Realism metrics also measures surface statistics, instead of semantic plausibility.

I also have some concern over the compute comparison and potential judge bias. Table 5's FLOPs column omits the massive upfront cost of calling Gemini3.1-Pro to label LESs labels, SFT intents, and RL intents plus fine-tuning Gemma3-4B-pt. What's more, Gemini3.1-Pro is used both to generate the training labels for LES and as the final judge in Section 4 (Critique F1) and Section 5 (WildBenchV2), which can subtly favor models distilled from Gemini3.1-Pro's preferences. It might be helpful to show experiments on final judge sensitivity.

Questions for authors

What are the RL performances of using newer models such as Qwen3 family or bigger models such as 4B or 8B+? Is there any particular reason for selecting old Qwen2.5-1.5B-Instruct as the main experiment model?

3 comments
AuthorsOpenReview ↗

Response to Reviewer vCZX

Thank you so much for your efforts reviewing our paper, we really appreciate your time and valuable suggestions! We respond to your concerns and questions below.

(1) RL results: We agree that the performance improvement of LES over the prompted simulators in RL were modest, as we have noted in Section 5.2. We clarify that the main benefit in RL comes from intrinsic rewards, which cut computational cost by more than half compared to an external judge, which holds for both types of simulators, with small performance improvements with the LES simulator.

(2) Single-turn GRPO comparison: Our main goal in the RL results of Section 5 is to compare multi-turn GRPO with different simulator setups. However, the comparison to single-turn GRPO is an interesting experiment! We performed a single-turn RL training run of Gemma3-4B-LES which achieved a score of 40.22 on WildBenchV2, compared to 41.44 achieved by the multi-turn training setup by the same simulator. We find the main benefit to be on the samples of the benchmark which consist of multi-turn contexts. We are happy to extend this analysis to the other RL training setups and include them in the paper.

(3) Human Study for Realism: We ran a Turing-style human study to measure realism on 150 samples SimulatorArena. Two annotators saw two candidate next user messages, one written by the real user and one generated by a simulator, and picked the one they thought was real. The fool rate for each simulator is summarized in the Table below. The annotators mistook LES messages for the real user’s 59% of the time, so they were indistinguishable from real users. The SFT simulator reached 48%, and Gemma3-4B-IT reached only 19%. The results of this human study are consistent with our statistical metrics and perplexity evaluations.

SimulatorFool Rate
Gemma3-4b-LES59.0%
Gemma3-4b-SFT48.0%
Gemma3-4b-IT19.0%

Judge Sensitivity: We re-ran the evaluation for the results on WildBenchV2 of Table 5 using a different judge LLM (Claude Haiku-5.5) instead of Gemini3.1-Pro and report the results below. We find similar trends in the results, where LES models still achieve the highest scores, which addresses the concern of a bias in the judge LLM. We also note that this evaluation is based on an explicit checklist rubric provided by the benchmark, which avoids judge biases.

SimulatorGemini3.1-ProClaude Haiku 5.5
—28.7825.39
Gemma3-4b-IT (Judge)39.7935.64
Gemma3-4b-IT (Intrinsic)40.5235.35
Gemma3-4b-SFT (Judge)40.5935.61
Gemma3-4b-LES (Judge)41.0836.10
Gemma3-4b-LES (Intrinsic)41.4436.40

Qwen assistant size: The multi-turn RL experiment is computationally expensive. Under our current computational budget and the specified GRPO hyperparameters (number of rollouts, turns, etc.), each RL training run on the WildChat-4.8M dataset took 2 weeks to finish training the Qwen2.5-1.5B model. This made it prohibitive to explore training larger models such as 4B and 8B+ models. We also opted for Qwen2.5 which offered a small enough size (1.5B) that we can train, and compare in a consistent fashion with the multi-turn RL training experiments of Suh et al., upon which we base our setup.

Thank you so much again for your review! We hope our additional results and clarifications address your concerns, and we would really appreciate it if you would increase your score.

Dear Reviewer vCZX,

We wanted to remind you that the discussion period ends tomorrow. We would appreciate it if you could take a moment to check our response and let us know if you have any additional questions or clarification requests.

Thank you for your time!

Thank the authors for their responses and I carefully read them. (Given the author claims improving performance is not the main goal) I'm now concerning more on the "computation cost cut" claim that I put in my original review, since there are still large upfront cost of calling proprietary models to label and prepare data. Having some detailed ablations over single-turn RL vs multi-step RL is also good to be included to make the paper clear about the performance gains. I believe another round of paper refine will make the paper stronger so i lean to keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.