Training User Simulators with Latent Intrinsic Feedback

User simulators are gaining attention as environments to train LLM assistants that can better cope with the nuances of multi-turn conversations with real users. Prior work on user simulation focuses primarily on generating realistic user utterances, leaving unexplored their potential as a source of direct feedback on the assistant's responses. Such an evaluation signal may not always be explicitly communicated in language, but can instead reside in the latent reasoning of users. In this work, we introduce user simulators with intrinsic feedback, which are models equipped with latent evaluation states: internal thoughts, reactions, and scalar ratings that evaluate the assistant from a first-person user perspective at each conversational turn. We train LLM-based simulators using interaction logs between real users and assistants and demonstrate that the use of intrinsic feedback improves simulation realism and behavioral alignment with real human users. Further, we show that evaluation signals generated by our trained simulators are better correlated with human judgments, compared to evaluations generated by standard prompted simulator baselines. Lastly, our experiments reveal that simulators with intrinsic feedback provide better reward signals for multi-turn RL training, producing more capable LLM assistants while simultaneously improving computational efficiency.