- Rating
- 1
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper highlights that existing simulation work focuses too heavily on surface-level fidelity (such as naturalness, mimicry, and behavioral coverage) under the implicit assumption that a realistic-looking user will yield valid downstream agent evaluations. To challenge this, the paper introduces outcome calibration: measuring whether simulators successfully reproduce human task success rates, conditional failure distributions, and paired session outcomes. To address outcome calibration, the authors propose Calibrated User Embeddings (CUE). CUE encodes an observed interaction into a continuous session representation, decodes it into natural-language persona commands, retrieves behavioral examples, and can also sample new representations using a latent diffusion model. The paper evaluates naturalness, user mimicry, behavioral coverage, aggregate success-rate agreement, paired outcome agreement, and failure-mode agreement. Experiments cover tau2-bench with 483 human episodes, SimulatorArena, and PRISM, using several simulator LLMs and baselines including UserLM, USP, PPol, RealUserSim, and a no-persona simulator. CUE is generally competitive on fidelity and often improves outcome agreement over other persona-based methods, although the no-persona baseline remains competitive in several settings.
Critical strengths
-
The paper identifies an important gap in current user-simulator evaluation. The distinction between surface fidelity and outcome calibration is convincing and consequential. The experiments show that methods that appear realistic under standard fidelity metrics can still substantially distort agent success rates or failure distributions. I view this evaluation framing as the strongest contribution of the paper.
-
Decoding latent representations into natural-language persona prompts rather than fine-tuning simulator weights avoids tying the simulator to a single open-weight backbone (unlike UserLM or USP) and allows zero-shot transfer across stronger proprietary LLMs. That makes the framework practically useful as simulator models improve.
Critical weaknesses
-
Complex pipeline, lack of ablations and unmatched baseline comparisons:
-
CUE combines LLM data labeling, two ModernBERT encoders with multiple representation and command-generation losses, a 15-slot Qwen3-0.6B decoder, a 162M-parameter diffusion sampler, and 8-shot example retrieval. The paper didn't test which of these parts actually drives the performance. For example, Table 16 shows that simply resampling directly from the training bank gets almost the same coverage as diffusion, but the authors never report outcome calibration for bank resampling.
-
CUE also has an unmatched conditioning advantage over the closest baselines. At test time, CUE adds 8 retrieved style examples to its simulator prompt, while RealUserSim is evaluated with all example utterances removed in the replay setting. Not sure if CUE's advantage over RealUserSim comes from its learned embeddings or simply from having 8 in-context examples and in-domain customer service data. A matched experiment giving baselines the same retrieval budget, together with CUE-without-retrieval / retrieval-only controls, would make the source of the gains much clearer.
-
PPol is omitted from sampled coverage and sampled outcome comparisons despite being a population-level persona generator (takes a task as input and generates N personas for it). The authors state that PPol is omitted from sampled coverage and outcome comparisons but do not explain why. Reporting population-level PPol results on the same tasks would provide a more appropriate comparison to CUE's sampled setting.
-
-
It is still unclear how much CUE improves benchmarking over a basic simulator, especially in the cross-agent setting that motivates the method.
-
On -Bench, a basic simulator with no persona (None) often ties or beats CUE on success-rate error and outcome F1 for Llama 3.1 8B and GPT-5.4 Mini. Also, it looks like 'None' gets its variance mainly from sampling noise, meaning CUE's improvements over a vanilla prompt might be very small or inconsistent.
-
The main practical motivation of reusing a user representation across agents is not exactly tested. On tau2-Bench, the replay uses the same agent family/setup as the human interaction. It would be more convincing to infer CUE from interactions with one agent, then test whether it preserves human-relative outcomes or agent rankings when interacting with a different agent.
-
Questions for authors
-
Missing SimulatorArena math tutoring outcomes? The abstract claims CUE generalizes to "document creation, math tutoring, and casual conversation", and notes that SimulatorArena contains 450 mathematics interactions. I see outcome calibration is reported only for the writing subset (Table 4), but couldn't locate the math tutoring results.
-
How would RealUserSim or a simpler profile baseline perform if given the same retrieved examples as CUE?
-
Why is PPol omitted from sampled coverage and sampled outcome comparisons, even when it is actually designed as a task-conditioned population-level persona generator?
-
Figure 1 shows 4 Stylistic slots, whereas a lot of places in the Appendix specify 5 style slots. Which configuration was trained?
3 comments
Fidelity and outcome calibration of the vanilla simulator.
We thank the reviewer for highlighting the competitive performance of the no-persona baseline. We believe this result reinforces a central finding of our paper: agreement on aggregate outcomes does not necessarily imply that a simulator faithfully represents the users producing those outcomes. Just as high naturalness does not guarantee outcome calibration, low-fidelity simulations can achieve competitive aggregate calibration scores. While the vanilla simulator performs comparably on some calibration metrics, it performs significantly worse on fidelity. This distinction matters for interactive benchmarking. A simulator can reproduce an aggregate success rate without reproducing the underlying distribution of user behaviors, and agreement at the aggregate level may conceal differences in which interactions succeed or fail and why. CUE is designed to address this broader objective: preserving competitive outcome alignment while substantially improving behavioral fidelity, without directly optimizing outcome labels. Rather than claiming that CUE uniformly dominates vanilla prompting on every calibration metric, our results demonstrate that it offers a more balanced combination of user fidelity and outcome calibration. This is precisely why we advocate evaluating these dimensions jointly rather than selecting simulators based on a single success-rate or fidelity measure.
SimulatorArena math tutoring outcomes
We thank the reviewer for pointing out this distinction. We evaluate fidelity on the SimulatorArena mathematics subset but do not report outcome calibration for these interactions because tutoring introduces an additional dimension: the user's knowledge may change during the interaction as a result of the agent's instruction. Evaluating outcomes in this setting would therefore require modeling the user's evolving knowledge state, rather than only their conversational behavior and task outcomes. This is outside the scope of the current work. We will clarify that our generalization results cover fidelity across document creation, mathematics tutoring, and casual conversation, whereas outcome calibration is evaluated only in settings where the relevant outcomes can be assessed without modeling changes in user knowledge.
PPol sampled comparisons.
PPol was excluded from sampled coverage and outcome comparisons because its persona-generation procedure was provided with the target real-user trajectory. Consequently, the generated persona is conditioned on an observed user interaction rather than sampled independently from a learned population distribution. Our sampled evaluation instead requires generating simulated users without access to the target real-user trajectory. Including PPol under its evaluated conditioning protocol would therefore conflate user-conditioned replay with population sampling. We agree that PPol's task-conditioned persona generation makes it a relevant population-level baseline. We will clarify that the omission reflects the conditioning protocol used in our implementation, rather than an inherent inability of PPol to generate sampled personas. A separately configured task-only PPol baseline would be an appropriate additional comparison, but this would be a modified variant of PPol rather than the method published.
Number of stylistic slots.
Thank you for identifying this inconsistency. The trained CUE checkpoint uses five stylistic slots. Figure 1 incorrectly depicts four. We will correct the figure to show five stylistic slots, consistent with the trained configuration and appendix description. We will add the new component ablations and paired uncertainty analysis to the appendix of the camera-ready version. We will also clarify the scope of our outcome-calibration claims, the distinction between simulator-base and cross-agent transfer, and the outstanding evaluation limitations identified above.
Thank you for the rebuttal and new experiments.
It seems like the new numbers actually show that CUE's heavy setup might not be very necessary. If CUE doesn't consistently beat the basic None model on outcome calibration, that weakens the main point of the paper (fidelity to outcome calibration).
Also, the new tests show RealUserSim gets a similar outcome F1 (.54 - .55) to CUE (.52 - .60) when given the same examples, suggesting that CUE's gains come from the retrieval, probably not from its complex model? Diffusion tests also show that simple bank resampling mostly matches or beats diffusion.
While the motivation is interesting, I think the method became complicated, and omitting a few metrics for some baselines also makes it hard to judge the benefits of the proposed method.
We thank the reviewer for recognizing outcome calibration as an important evaluation problem and the practical advantages of model-agnostic user conditioning. We agree that the original submission would benefit from more ablations and clearer discussion of the limits of our calibration claims. To address these concerns, we conducted additional evaluations on Tau2-Bench on GPT 5.4 Mini and Gemini 3.5 Flash Lite (Llama was omitted due to time constraint).
Unmatched baselines and component ablations.
Since CUE’s eight retrieved examples might introduce an asymmetry relative to RealUserSim, we evaluated CUE as in the paper (CUE Commands + CUE examples), CUE with commands only, CUE with examples only, and RealUserSim with the same CUE-retrieved examples (RealUserSim commands + CUE examples) while retaining its own persona generation procedure.
CUE retains a naturalness advantage over retrieval-matched RealUserSim on both bases. We additionally trained CUE without auxiliary representation-learning losses and CUE without the command-slot architecture.
Removing representation losses decreases S2R-N by 0.076 [0.053, 0.098] on GPT and 0.093 [0.067, 0.119] on Gemini. Removing command slots decreases it by 0.056 [0.033, 0.080] and 0.079 [0.052, 0.107], respectively. All four paired intervals exclude zero. These findings support the contribution of the learned architecture to fidelity.
Lastly, we now report both fidelity and outcome calibration for uniform training bank resampling (sampling persona prompts from the training set as generated using the contrastive method) and neighborhood diffusion (sampling using the diffusion sampler while providing 8 dataset samples to direct the sampler to the same user space as the original dataset) in addition to the unconditional diffusion sampling from the original paper.
Diffusion improves sampled naturalness relative to uniform bank sampling on both bases: +0.096 [0.067, 0.124] on GPT and +0.145 [0.111, 0.179] on Gemini. However, calibration differences are mixed, and the paired intervals for the reported sampled calibration metrics include zero. We therefore interpret diffusion primarily as improving sampled fidelity and coverage, rather than as uniformly improving outcome calibration.