- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes user simulators with latent evaluation states that generate both user responses and turn-level intrinsic feedback on the assistant. The simulator is trained on WildChat interactions with synthetic LES annotations and is further used to provide reward signals for multi-turn RL, reducing the need for an external judge model.
Critical strengths
- The idea of user simulators with intrinsic feedback is interesting and practically useful, allowing the simulator to provide turn-level evaluations directly during multi-turn RL.
- The experiments are relatively comprehensive, covering multiple model scales and families, feedback behavior, simulation realism, human rating correlation, ablations, and multi-turn RL.
Critical weaknesses
- The LES supervision is generated by Gemini3.1-Pro. The paper does not report human verification or agreement analysis for these synthetic labels, leaving their quality unclear.
- Most annotated turns are Neutral, yet Internal Thought and Rating are still generated for these turns. Since these latent evaluations are not directly observed from users, it remains unclear how reliably they reflect the actual user state.
- The benefits of full LES training are not consistent across evaluations. Standard SFT achieves higher Critique F1 across all Gemma scales, the Rating-only ablation achieves higher F1 than full LES, and the gains in human rating correlation are not consistently significant across model scales and subsets.
Questions for authors
- Could the authors provide human validation or agreement analysis for the generated LES labels, especially Internal Thought and Rating for Neutral turns?
- Related work such as ThoughtTrace collects direct self-reported thoughts and finds that latent user states are difficult for frontier LLMs to infer from conversation context. Could the authors further validate whether the Gemini-generated LESs reflect direct user evidence?
2 comments
Dear Reviewer gBvD,
We wanted to remind you that the discussion period ends tomorrow. We would appreciate it if you could take a moment to check our response and let us know if you have any additional questions or clarification requests.
Thank you for your time!
Response to Reviewer gBvD
Thank you so much for your efforts reviewing our paper, we really appreciate your time and valuable suggestions! We respond to your concerns and questions below.
(1) Human Verification of LES labels: We conducted a human annotation of the LES for 100 randomly sampled conversations, with 20 conversations sampled from each rating level. The three human annotators achieved a good average Pearson correlation of 0.74 with the Gemini-generated labels. We hope this provides further quality guarantees about our approach for synthetic labeling
(2) Clarification on Neutral turns: We considered turns that are not explicit praises or critiques as “Neutral”, but consist of other reactions such as follow-ups, re-prompting, or topic shifting, for which the ratings can vary. We agree that the term Neutral may not be the best terminology here and suggest revising it to “Other” reactions for less confusion.
(3) Clarification on full LES benefits: We clarify that the training ablations in Table 2 were conducted to better understand the impact of the different LES components. We find that full LES training is primarily beneficial for improving simulation realism compared to SFT models, which is a crucial aspect of simulator quality. However, this comes at the cost of a lower critique F1 compared to training with ratings alone, which achieves the strongest critiquing ability and even outperforms the SFT model. Given this tradeoff, we see the main benefit of full LES training as combining strong evaluative capabilities, comparable to those of prompted simulators, with substantially better simulation realism. We have revised our discussion of these ablation results in Section 4.3 to better reflect this finding.
(4) Comparison to ThoughtTrace: We clarify that in ThoughtTrace, the self-reported thoughts of the human users were completely open-ended, which indeed makes it difficult for LLMs to infer. In contrast, the thoughts in our LES are scoped to evaluate whether the assistant followed the user’s instructions, based on the conversation history and what the next user turn was, which is relatively easier for LLMs to infer. The strong agreement levels of our human verification also supports this.
Thank you so much again for your review! We hope our additional human verification and clarifications addressed your concerns, and we would really appreciate it if you would increase your score.