- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces RUSTLeD, a framework for training LLM-based user simulators with natural-language latent states. It adopts an EM-style approach in which an LLM labeler proposes latent-state trajectories conditioned on complete observed conversations. The labeler’s distribution serves as a semantic anchor for posterior-regularized marginal-likelihood maximization. A self-normalized importance-sampling procedure reweights the proposed trajectories, and the simulator is updated through weighted supervised fine-tuning. Experiments on ConvApparel-V2 evaluate generalization to held-out shopping-assistant personas under two policy splits. The results show benefits over prompted simulators and improvements over direct SFT without latent states on several behavioral metrics.
Critical strengths
-
The paper studies an interesting and useful problem. Reliable user simulators could reduce the cost of collecting human interactions and help develop and evaluate conversational agents. Making these simulators work well with previously unseen agent policies is an important research direction.
-
The method is well motivated and produces promising results. The EM-style framework uses LLM-generated latent states as proposals rather than fixed ground-truth labels. It reweights these proposals based on the simulator’s likelihood over the dialogue and latent states, then updates the simulator via weighted SFT. On ConvApparel-V2, the method improves several aspects of simulated behavior compared with prompting and direct SFT without latent states, including repetition and goal-satisfaction calibration.
Critical weaknesses
-
The labeler, simulator, and judge come from the same model family. The labeler uses Gemini 2.5 Pro, while the simulator and judge use Gemini 2.5 Flash. These models may share biases about what realistic user behavior looks like. This raises concerns about whether the reported gains would hold under an independent evaluation.
-
The main behavioral metrics lack human validation. The paper uses real human conversations as references and reports some deterministic metrics. However, dialogue acts, goal satisfaction, and human-likeness are assessed by an LLM judge without checking its agreement with human annotators. Since the goal is to simulate human users, a human-annotated subset would help establish whether the judge’s assessments and method rankings are reliable.
-
Downstream evaluation would strengthen the practical value. It would be useful to test whether agents trained or optimized with this simulator perform better with real users.
Questions for authors
-
How do you assess whether the learned latent states are reasonable? I understand that users’ true internal states are not observed. Have you checked whether human annotators find the generated states plausible and consistent with the conversations, or whether they change substantially with a different labeler? This would help establish whether the states offer meaningful explanations of user behavior.
-
Do the judge’s assessments agree with human judgments? Have you evaluated this on a subset of conversations? In particular, do humans reach similar conclusions about the relative quality of the simulators?
6 comments
Response to Reviewer MQbA (2/2)
3. Assessing whether learned latent states are reasonable & sensitivity to the labeler (Q1)
- Instrumental role in HMLLMs. Because shoppers' true internal states are unobserved, RUSTLeD treats instrumentally — as a latent state variable whose role is to carry predictive causal structure from the history to subsequent user utterances, applicable to any stochastic labeler . SNIS-EM enforces this directly: Appendix D.2 shows that the log importance weight decomposes into emission predictive power minus causal implausibility, penalizing plausible-sounding annotations that fail to explain what the user actually says.
- Design and empirical sensitivity to the labeler. Appendix C details our two-stage annotation schema: a -field structured state (intent, constraints, affect, trust, patience reserve, task outlook, etc.) paired with a strictly prefix-causal first-person reaction (
Impact), while hindsightEvidencecitations are stripped before training so future text never enters 's conditioning set. Sensitivity to the labeler is directly tested by our Smoother vs. Filtering ablation (Section 4.2.4): using the exact sameGemini 2.5 Promodel and -field schema, changing only prefix vs. full-conversation conditioning leaves teacher-forced utterance likelihood unchanged ( vs. ) while cutting ( to pp) and verbatim repetition ( to ). Thus closed-loop rollout behavior genuinely depends on the quality of the learned dynamics. - Inspecting generated latent states. In Appendix G (Qualitative Examples), we omitted the simulator's generated blocks for space; in the revision, we will display the generated latent states alongside the utterances so readers can inspect their plausibility directly.
4. Downstream evaluation with real users (W3)
We agree that training or selecting assistant policies against the simulator and validating them in live human A/B studies is an important next step, though not one that is feasible for us at this time. We note two points:
- Closed-loop evaluation across unseen assistant policies with real human interaction logs from engagement with these policies (
ConvApparel-V2) is already a direct multi-policy off-policy evaluation (OPE) test: each unseen persona represents a distinct candidate agent policy absent from training, and measuring , , and against real humans interacting with those exact policies tests whether the simulator accurately predicts policy-induced changes in human outcomes: this is where SFT (No LS) overshoots human satisfaction by pp and Filtering overshoots it by pp. By contrast, we see a difference of only pp for RUSTLeD / Smoother. - End-to-end RL policy training across multiple simulator baselines followed by live human-subject evaluation constitutes a standalone human-study project beyond the scope of this paper. We will highlight this opportunity clearly in Future Work.
Thank you for the response. I appreciate the effort, but I believe my main concerns remain.
- When changing the judge model, the results in Table C4 are mixed. On 5→6, SFT (No LS) has the lowest L1_act (0.410), lower than all RUSTLeD settings (0.465–0.754). On 3→8, RUSTLeD (M=15) has a |ΔSat| of 9.8–10.9 pp, compared with 4.3 pp for SFT (No LS). The two judges also disagree on the human reference itself (Sat. 20.7% vs. 10.5%). From my understanding, this suggests that the method ranking depends on the judge. In addition, the labeler and simulator are still from the same family.
- Agreement between two LLM judges does not replace agreement with human annotators, and this agreement is not strong (Cohen’s κ = 0.48–0.53 for goal satisfaction). A small human-annotated subset would be enough to check the judge.
- My question about the latent states is not answered. The Smoother vs. Filtering comparison uses the same labeler model with a different input, so it does not show how the states change with a different labeler, or whether humans find them plausible.
- I understand that a live human study is not feasible during the rebuttal. However, the existing settings already allow a easy check: whether the simulator ranks the unseen assistants in the same order as real users. This would give some evidence of downstream value.
Since these points are not resolved, I am inclined to maintain my current score.
Response to Reviewer MQbA (1/2)
Thank you for the constructive review and clear questions. We address the weakness points raised and questions below.
1. Shared model family across labeler, simulator, and judge (W1)
We address this above in the common response (Table C4).
2. Reliability of behavioral metrics and agreement with human / independent judgments (W2 & Q2)
Our primary behavioral metrics (L1act=∑a∣rm(a)−rhum(a)∣ and ∣ΔSat∣=∣Satm−Sathum∣) are much less sensitive to judge leniency than direct pairwise LLM preferences: both real human dialogues and simulated dialogues are labeled with precisely the same temperature-0 categorical rubrics (Appendix F.2), so any constant bias cancels unless the judge shifts its rubric boundary between human and simulated text.
Across
3,072(5→6) and3,327(3→8) paired dialogues, Gemini 2.5 Flash and OpenAI GPT-5.4 Mini show high direct rubric agreement across all three LLM-judged constructs in Table 1 (plus Appendix FViol.%) on both the Real Human reference conversations and simulator rollouts:3,072convs)3,327convs)0.683(Real Humans0.711)0.684(Real Humans0.711)87.6%(Cohen's κ=0.53;Humans84.4%)89.0%(Cohen's κ=0.48;Humans85.5%)Sat.%+0.97 / +0.9389.9%within ±1 pt (Humans89.7%)90.0%within ±1 pt (Humans91.1%)Hum.+0.68 / +0.62(For Appendix F rubrics, rare
Viol.%achieves96.3%/96.7%exact agreement [96.5% / 96.1%onReal Humans];Push.has a constant~0.9-pt stricter scale offset on GPT-5.4 Mini [3.18vs.4.06], giving67.8% / 65.7%uncentered within-±1-pt agreement; 3→8 ρ spans the 13 fine-tuned models.)Running a new human-annotator labeling campaign over thousands of multi-turn dialogues was not feasible within the rebuttal window. We will include the cross-family inter-judge agreement analysis and an explicit discussion of human-annotation validation in the revised paper.