acceptodds
Under review as a conference paper at ICLR 2027

CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking

Abstract

Recent benchmarks rely on user simulators to evaluate AI agents in multi-turn interaction. While existing simulation techniques demonstrate surface fidelity to human style and behavior, ecologically valid interactive benchmarking also requires alignment in when and how agents fail across simulated and real user populations. We find that established measures of simulator naturalness and mimicry provide inconsistent signals about whether simulated interactions reproduce real-user outcomes. Therefore, interactive benchmarking requires a complementary criterion, outcome calibration: agreement with observed success rates, conditional failure patterns, and retrospective user-task outcomes. We introduce Calibrated User Embeddings (CUE), a framework that encodes observed sessions or samples continuous representations, then decodes them into persona commands augmented with retrieved examples to steer a black-box simulator LLM. Through this, we evaluate both user-conditioned replay of past sessions and aggregate metric agreement when sampling novel personas for the same tasks. On -Bench, CUEd simulators commit fewer simulator-attributed errors and more faithfully reproduce real-user agent failure modes, aggregate success rates, and outcomes for specific task-user pairs than other persona-based simulation methods. These gains coexist with competitive user fidelity as measured using metrics established in prior work. After being fit to mostly customer support interactions, the same CUEd simulators generalize to document creation, math tutoring, and casual conversation, and remain effective across different simulator LLMs without CUE retraining.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
user simulation, multi-turn, benchmarking, evaluation, prompt generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_9v1gOpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper highlights that existing simulation work focuses too heavily on surface-level fidelity (such as naturalness, mimicry, and behavioral coverage) under the implicit assumption that a realistic-looking user will yield valid downstream agent evaluations. To challenge this, the paper introduces outcome calibration: measuring whether simulators successfully reproduce human task success rates, conditional failure distributions, and paired session outcomes. To address outcome calibration, the authors propose Calibrated User Embeddings (CUE). CUE encodes an observed interaction into a continuous session representation, decodes it into natural-language persona commands, retrieves behavioral examples, and can also sample new representations using a latent diffusion model. The paper evaluates naturalness, user mimicry, behavioral coverage, aggregate success-rate agreement, paired outcome agreement, and failure-mode agreement. Experiments cover tau2-bench with 483 human episodes, SimulatorArena, and PRISM, using several simulator LLMs and baselines including UserLM, USP, PPol, RealUserSim, and a no-persona simulator. CUE is generally competitive on fidelity and often improves outcome agreement over other persona-based methods, although the no-persona baseline remains competitive in several settings.

Critical strengths
  • The paper identifies an important gap in current user-simulator evaluation. The distinction between surface fidelity and outcome calibration is convincing and consequential. The experiments show that methods that appear realistic under standard fidelity metrics can still substantially distort agent success rates or failure distributions. I view this evaluation framing as the strongest contribution of the paper.

  • Decoding latent representations into natural-language persona prompts rather than fine-tuning simulator weights avoids tying the simulator to a single open-weight backbone (unlike UserLM or USP) and allows zero-shot transfer across stronger proprietary LLMs. That makes the framework practically useful as simulator models improve.

Critical weaknesses
  • Complex pipeline, lack of ablations and unmatched baseline comparisons:

    • CUE combines LLM data labeling, two ModernBERT encoders with multiple representation and command-generation losses, a 15-slot Qwen3-0.6B decoder, a 162M-parameter diffusion sampler, and 8-shot example retrieval. The paper didn't test which of these parts actually drives the performance. For example, Table 16 shows that simply resampling directly from the training bank gets almost the same coverage as diffusion, but the authors never report outcome calibration for bank resampling.

    • CUE also has an unmatched conditioning advantage over the closest baselines. At test time, CUE adds 8 retrieved style examples to its simulator prompt, while RealUserSim is evaluated with all example utterances removed in the replay setting. Not sure if CUE's advantage over RealUserSim comes from its learned embeddings or simply from having 8 in-context examples and in-domain customer service data. A matched experiment giving baselines the same retrieval budget, together with CUE-without-retrieval / retrieval-only controls, would make the source of the gains much clearer.

    • PPol is omitted from sampled coverage and sampled outcome comparisons despite being a population-level persona generator (takes a task as input and generates N personas for it). The authors state that PPol is omitted from sampled coverage and outcome comparisons but do not explain why. Reporting population-level PPol results on the same tasks would provide a more appropriate comparison to CUE's sampled setting.

  • It is still unclear how much CUE improves benchmarking over a basic simulator, especially in the cross-agent setting that motivates the method.

    • On τ2\tau^2-Bench, a basic simulator with no persona (None) often ties or beats CUE on success-rate error and outcome F1 for Llama 3.1 8B and GPT-5.4 Mini. Also, it looks like 'None' gets its variance mainly from sampling noise, meaning CUE's improvements over a vanilla prompt might be very small or inconsistent.

    • The main practical motivation of reusing a user representation across agents is not exactly tested. On tau2-Bench, the replay uses the same agent family/setup as the human interaction. It would be more convincing to infer CUE from interactions with one agent, then test whether it preserves human-relative outcomes or agent rankings when interacting with a different agent.

Questions for authors
  1. Missing SimulatorArena math tutoring outcomes? The abstract claims CUE generalizes to "document creation, math tutoring, and casual conversation", and notes that SimulatorArena contains 450 mathematics interactions. I see outcome calibration is reported only for the writing subset (Table 4), but couldn't locate the math tutoring results.

  2. How would RealUserSim or a simpler profile baseline perform if given the same retrieved examples as CUE?

  3. Why is PPol omitted from sampled coverage and sampled outcome comparisons, even when it is actually designed as a task-conditioned population-level persona generator?

  4. Figure 1 shows 4 Stylistic slots, whereas a lot of places in the Appendix specify 5 style slots. Which configuration was trained?

3 comments

We thank the reviewer for recognizing outcome calibration as an important evaluation problem and the practical advantages of model-agnostic user conditioning. We agree that the original submission would benefit from more ablations and clearer discussion of the limits of our calibration claims. To address these concerns, we conducted additional evaluations on Tau2-Bench on GPT 5.4 Mini and Gemini 3.5 Flash Lite (Llama was omitted due to time constraint).

Unmatched baselines and component ablations.

Since CUE’s eight retrieved examples might introduce an asymmetry relative to RealUserSim, we evaluated CUE as in the paper (CUE Commands + CUE examples), CUE with commands only, CUE with examples only, and RealUserSim with the same CUE-retrieved examples (RealUserSim commands + CUE examples) while retaining its own persona generation procedure.

MethodGPT S2R-NGemini S2R-NGPT Outcome F1Gemini Outcome F1
CUE commands + CUE examples0.510.650.600.52
CUE commands0.520.580.560.54
CUE examples0.310.570.580.56
RealUserSim commands0.330.360.550.57
RealUserSim commands + CUE examples0.370.490.540.55

CUE retains a naturalness advantage over retrieval-matched RealUserSim on both bases. We additionally trained CUE without auxiliary representation-learning losses and CUE without the command-slot architecture.

MethodGPT S2R-NGemini S2R-NGPT Outcome F1Gemini Outcome F1
CUE0.510.650.600.52
CUE w/o representation losses0.440.560.560.54
CUE w/o command slots0.460.570.540.54
Direct contrastive manuals0.490.630.600.54

Removing representation losses decreases S2R-N by 0.076 [0.053, 0.098] on GPT and 0.093 [0.067, 0.119] on Gemini. Removing command slots decreases it by 0.056 [0.033, 0.080] and 0.079 [0.052, 0.107], respectively. All four paired intervals exclude zero. These findings support the contribution of the learned architecture to fidelity.

Lastly, we now report both fidelity and outcome calibration for uniform training bank resampling (sampling persona prompts from the training set as generated using the contrastive method) and neighborhood diffusion (sampling using the diffusion sampler while providing 8 dataset samples to direct the sampler to the same user space as the original dataset) in addition to the unconditional diffusion sampling from the original paper.

MethodGPT S2R-NGemini S2R-NGPT Agent TVD ↓Gemini Agent TVD ↓GPT Rate |Δ|Gemini Rate |Δ|
Unconditional diffusion0.560.740.280.290.090.14
Uniform training bank resampling0.470.590.250.400.080.14
Neighborhood diffusion0.490.630.330.380.120.15

Diffusion improves sampled naturalness relative to uniform bank sampling on both bases: +0.096 [0.067, 0.124] on GPT and +0.145 [0.111, 0.179] on Gemini. However, calibration differences are mixed, and the paired intervals for the reported sampled calibration metrics include zero. We therefore interpret diffusion primarily as improving sampled fidelity and coverage, rather than as uniformly improving outcome calibration.

Fidelity and outcome calibration of the vanilla simulator.

We thank the reviewer for highlighting the competitive performance of the no-persona baseline. We believe this result reinforces a central finding of our paper: agreement on aggregate outcomes does not necessarily imply that a simulator faithfully represents the users producing those outcomes. Just as high naturalness does not guarantee outcome calibration, low-fidelity simulations can achieve competitive aggregate calibration scores. While the vanilla simulator performs comparably on some calibration metrics, it performs significantly worse on fidelity. This distinction matters for interactive benchmarking. A simulator can reproduce an aggregate success rate without reproducing the underlying distribution of user behaviors, and agreement at the aggregate level may conceal differences in which interactions succeed or fail and why. CUE is designed to address this broader objective: preserving competitive outcome alignment while substantially improving behavioral fidelity, without directly optimizing outcome labels. Rather than claiming that CUE uniformly dominates vanilla prompting on every calibration metric, our results demonstrate that it offers a more balanced combination of user fidelity and outcome calibration. This is precisely why we advocate evaluating these dimensions jointly rather than selecting simulators based on a single success-rate or fidelity measure.

SimulatorArena math tutoring outcomes

We thank the reviewer for pointing out this distinction. We evaluate fidelity on the SimulatorArena mathematics subset but do not report outcome calibration for these interactions because tutoring introduces an additional dimension: the user's knowledge may change during the interaction as a result of the agent's instruction. Evaluating outcomes in this setting would therefore require modeling the user's evolving knowledge state, rather than only their conversational behavior and task outcomes. This is outside the scope of the current work. We will clarify that our generalization results cover fidelity across document creation, mathematics tutoring, and casual conversation, whereas outcome calibration is evaluated only in settings where the relevant outcomes can be assessed without modeling changes in user knowledge.

PPol sampled comparisons.

PPol was excluded from sampled coverage and outcome comparisons because its persona-generation procedure was provided with the target real-user trajectory. Consequently, the generated persona is conditioned on an observed user interaction rather than sampled independently from a learned population distribution. Our sampled evaluation instead requires generating simulated users without access to the target real-user trajectory. Including PPol under its evaluated conditioning protocol would therefore conflate user-conditioned replay with population sampling. We agree that PPol's task-conditioned persona generation makes it a relevant population-level baseline. We will clarify that the omission reflects the conditioning protocol used in our implementation, rather than an inherent inability of PPol to generate sampled personas. A separately configured task-only PPol baseline would be an appropriate additional comparison, but this would be a modified variant of PPol rather than the method published.

Number of stylistic slots.

Thank you for identifying this inconsistency. The trained CUE checkpoint uses five stylistic slots. Figure 1 incorrectly depicts four. We will correct the figure to show five stylistic slots, consistent with the trained configuration and appendix description. We will add the new component ablations and paired uncertainty analysis to the appendix of the camera-ready version. We will also clarify the scope of our outcome-calibration claims, the distinction between simulator-base and cross-agent transfer, and the outstanding evaluation limitations identified above.

Thank you for the rebuttal and new experiments.

It seems like the new numbers actually show that CUE's heavy setup might not be very necessary. If CUE doesn't consistently beat the basic None model on outcome calibration, that weakens the main point of the paper (fidelity to outcome calibration).

Also, the new tests show RealUserSim gets a similar outcome F1 (.54 - .55) to CUE (.52 - .60) when given the same examples, suggesting that CUE's gains come from the retrieval, probably not from its complex model? Diffusion tests also show that simple bank resampling mostly matches or beats diffusion.

While the motivation is interesting, I think the method became complicated, and omitting a few metrics for some baselines also makes it hard to judge the benefits of the proposed method.

Reviewer_1KyeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a system composed of representation learning and LLM prompting, so that the prompted LLM acts as a simulated user. For the former, a continuous representation of human conversational behavior is learnt. For the latter, the aforementioned embedding, the CUE, is used to generate natural language prompts that tells an LLM how to behave as a simulate user. To learn CUE, conversation transcripts are used to train a transformer-based neural model. To generate prompts from CUE, a language model with additional attention mechanism decodes it. Additionally, the proposed system also retrieves examples from related or adjacent training examples. A contrastive mechanism (real conversations versus generic simulator responses) is set up to encourage the learning behavior to align more with how real humans talk (through writing). During the generation process, an latent diffusion approach is used. The reported results suggest that under the proposed "fidelity" metric, the system often outperforms baselines. Transfer experiments suggest that the learning outcome is applicable to held-out domains not seen during the trainings.

Critical strengths

The paper's motivation is convincing. Understanding how LLMs could more convincingly simulate human behavior has a range of applications, from commercial settings like customer service to academic situations like doing social simulation. The authors also provide careful examination of potential harm from mis-using LLMs that could mimic humans in the ethics statement. The paper's distinction of fidelity and calibration makes sense. Superficial assessment of whether an LLM talks like a human being could conceal misalignments. The paper's use of representation learning, including contrastive learning, and diffusion mechanism for the proposed CUE embedding, as part of the LLM system presented, is reasonable. The paper's experiments assess multiple LLMs, and also tests transfer beyond the training setting. This is helpful because LLM behaviors could vary a lot, and that different use cases can warrant different behaviors.

Critical weaknesses

It could be helpful to understand the role of the proposed mechanisms (e.g. the continuous representation design, diffusion mechanism, contrastive setting) behind the observed performance gains through component-wise ablation. It may be helpful to understand how performances might differ under the same supervision. For instance, would the method perform better or worse if likewise to some baselines currently reported, it does not have the eight extra examples (that was retrieved)? It might be nice to further clarify advantages against more "naive" alternatives, such as giving a well-written prompt to a "smarter" LLM (e.g. more advanced model, more compute budget). For example, one might argue that with advanced reasoning capabilities and priors developed on a much larger corpus of text, such an LLM might be able to get more out of the comparatively limited set of conversation traces than training a bespoke neural model. (Less urgent, but I feel this is still noteworthy) It appears that the example responses, such as "ok . thanks . that works ." provided in the appendix (part H.2) are quite terse and have unusual formatting (space between commas, no capitalization). It could be said that from such short snippets, even a human might have trouble pinning down what type of behavior (which the proposed method aims to model) drives them. Potential anonymity issue: it would be helpful to ensure that the provided/linked materials do not contain the identifying author informations

Questions for authors

It appears that in the contrastive training setup, the same set of models (Llama, GPT, Gemini) is used to generate the "negative examples" i.e. the generic simulator responses. Thus, conceptually, it seems that the training is encouraging learning outcomes that avoids how these model behaves by default and move toward real human responses (the "positive examples.") If this is correct, how might alternative methods to achieve similar effects, such as LoRA finetuning directly on the real human responses perform differently? Relatedly, to what extent might a much smarter model be able to produce prompts on par with the propose pipeline's CUE-based generative process? This may be relevant to the paper's contribution item regarding CUE's capacity to support both replaying a known user and emulating a new user -- it could be said that an LLM could already do this, albeit to verying It could be helpful to understand what "continuous" means for CUE operationally. For instance, does "discrete versus continuous" refer to using prompts versus neural embeddings (whose weights are continuous/float datatypes), or are we comparing to embeddings that are innately discrete (e.g. token-based)? To what extent does sampling a new embedding from the CUE setup compare to sampling a new person? If the learnt distribution does not match that of the people we aim to simulate, what disparity might we observe?

3 comments

We thank the reviewer for recognizing the motivation between distinguishing fidelity from outcome calibration, as well as the value of evaluating simulator bases and held-out domains. We agree that the original submission would benefit from comparisons against simpler baselines, particularly direct prompting with stronger LLMs, and more detailed component ablations. We conducted additional experiments on Tau2-Bench using GPT 5.4 Mini and Gemini 3.5 Flash as simulator bases (Llama was omitted due to time constraints).

Stronger LLM prompting baseline.

We agree that a strong LLM may be able to infer behavioral instructions directly from conversational traces without requiring a specialized representation model. To investigate this, we evaluate two alternative methods:

  1. Direct contrastive manuals: Prompts generated directly using the contrastive annotation procedure employed to produce CUE’s training supervision.
  2. Fable 5.2 trajectory personas: Prompts and examples generated directly from the target human trajectory using a stronger reasoning model.

These methods bypass CUE’s learned representation pipeline.

MethodGPT S2R-NGemini S2R-NGPT AVAGemini AVA
CUE0.510.650.610.64
Direct Contrastive Manuals0.490.630.680.70
Fable 5.2 Personas0.530.820.800.92
MethodGPT Agent TVD ↓Gemini Agent TVD ↓GPT Outcome F1Gemini Outcome F1
CUE0.330.270.600.52
Direct Contrastive Manuals0.300.360.600.54
Fable 5.2 Personas0.300.350.530.58

These results demonstrate that stronger direct prompting is a competitive alternative, particularly for fidelity. Fable substantially improves mimicry over CUE, with paired AVA differences of +0.194

+0.138,+0.251+0.138, +0.251

on GPT and +0.275

+0.230,+0.321+0.230, +0.321

on Gemini.

However, higher mimicry does not consistently translate into better outcome calibration. On GPT, Fable's outcome F1 is lower than CUE's by 0.063 [0.010, 0.115], while on Gemini its observed improvement of 0.062 has a confidence interval approximately touching zero.

This provides additional evidence for the paper's central distinction between fidelity and calibration: a stronger LLM can produce highly convincing user personas without necessarily reproducing human-agent outcomes more accurately.

The direct contrastive-manual baseline is also competitive. Its primary disadvantage is the cost of generating contrastive annotations separately for new users, whereas CUE amortizes this process through a trained encoder–decoder. We will clarify this tradeoff rather than claim that learned representations universally outperform direct prompting.

Component-wise ablations.

We conducted additional inference and training ablations to isolate the contributions of CUE’s components.

MethodGPT S2R-NGemini S2R-NGPT Outcome F1Gemini Outcome F1
CUE0.510.650.600.52
Commands Only0.520.580.560.54
Examples Only0.310.570.580.56
No representation losses0.440.560.560.54
No command slots0.460.570.540.54

Removing the auxiliary representation-learning losses reduces S2R-N by 0.076 [0.053, 0.098] on GPT and 0.093 [0.067, 0.119] on Gemini (95% paired task-bootstrap CIs for the reduction). Removing the slot-based architecture produces reductions of 0.056 [0.033, 0.080] and 0.079 [0.052, 0.107], respectively. These findings support the contribution of the learned representation objectives and architecture to naturalness.

The commands-only and examples-only controls further demonstrate that CUE's performance cannot be attributed exclusively to retrieved examples. In particular, removing commands substantially reduces naturalness on GPT, while removing examples has a larger effect on Gemini.

Diffusion sampling vs. bank sampling.

We additionally compare uniform training bank resampling (sampling persona prompts from the training set as generated using the contrastive method) and neighborhood diffusion (sampling using the diffusion sampler while providing 8 dataset samples to direct the sampler to the same user space as the original dataset) in addition to the unconditional diffusion sampling from the original paper.

MethodGPT S2R-NGemini S2R-NGPT S2R-CGemini S2R-C
Unconditional diffusion0.560.740.850.86
Uniform training bank resampling0.470.590.830.82
Neighborhood diffusion0.490.630.840.84

Unconditional diffusion improves naturalness over bank resampling by +0.096 [0.067, 0.124] on GPT and +0.145 [0.111, 0.179] on Gemini. It also modestly improves S2R-C, while SD-C remains broadly comparable.

Calibration differences between diffusion and bank sampling are mixed. Thus, diffusion's primary demonstrated advantage is improved sampled fidelity and coverage, rather than a large improvement in outcome calibration.

What does continuous mean operationally?

By continuous, we mean that user behavior is represented as a point in a learned real-valued embedding space, rather than exclusively as a discrete set of persona attributes or a fixed textual profile. The distinction is not simply that neural network parameters are floating-point values. The value of this is that operations like sampling can be done in a more sound manner, leveraging factors like population-level likelihood and co-occurance to create more plausible unseen users when compared to bank resampling. Other unseen-user sampling methods, like that used in USP, combine parts of personas from different bank users, which may result in unlikely combinations.

Unnaturally formatted example responses.

We agree that short utterances with unconventional punctuation or capitalization can be difficult to interpret individually. These examples reflect the formatting of the underlying conversational data, and a single short response is not necessarily sufficient to infer a reliable behavioral characteristic. CUE is intended to learn from patterns across dialogue rather than infer a complete user profile from one isolated utterance. We will clarify this distinction in the appendix and improve the presentation of the qualitative examples, including providing additional conversational context where appropriate.

Anonymity concerns.

We thank the reviewer for flagging the potential anonymity concern. We have audited the manuscript, supplementary materials, and linked resources and ensured that they comply with the venue's anonymity requirements.

The new experiments clarify three findings: (1) stronger direct prompting can outperform CUE on mimicry without consistently improving outcome calibration; (2) CUE's representation objectives and slot-based architecture contribute to naturalness, although their individual calibration benefits remain less conclusive; and (3) diffusion improves sampled fidelity and coverage relative to reusing training-bank embeddings.

We will incorporate these results and clarify the distinction between continuous behavioral representations, direct prompting, and response-level fine-tuning in the revised manuscript, expanding what is above to include multiple seed runs and Llama results.

Thank you for the clarifications and additional experiments. This is a lot of work!

From the results, I can see the impact of the components such as commands, retrieved examples, and the diffusion setup behind the performance gains. For instance, it's helpful to see that the gains are not just due to having more examples.

Meanwhile, it appears that for calibration, CUE saw wins on GPT but losses on Gemini. For the custom representation loss setup and the slots, it seems that Gemini also seems to challenge the trend of decreasing F1 scores.

Additionally, it is helpful to see the current frontier models' attempt at the problem CUE aims to solve. I feel that it makes sense to explore different cost tradeoffs rather than defaulting to always using a smart model, and it would be helpful to understand the impact of that tradeoff. For example, when do we know it is worthwhile to bring out Fable instead of CUE?

For the effectiveness of diffusion on the reuse of training example, I can see that this choice is helpful for naturally, but again for calibration, which seems to be an important part of the authors claim, the pattern is less clear.

Overall, it appears that defaulting to a SOTA method does have a meaningful advantage over the proposed CUE approach, and that it appears CUE's interaction notably differs when we change the model (e.g. from GPT to Gemini). Considering these challenges, I'm afraid I would not be able to raise my rating.

For future work, I feel it might be useful to distinguish which parts are readily or effectively handled by a model like Fable, so that further investigation could be done on what Fable could not do.

Lastly, I would like to re-iterate the potental anonymity concern. It appears that the paper's linked material still leads to a webpage where the authors' names are identified. Though I have kept my assessment of the paper's scientific contribution separate from this concern, I feel that it would be helpful to keep the submission's content anonymous.

Reviewer_Avs9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether LLM-based user simulators reproduce human-grounded agent outcomes, arguing that behavioral fidelity alone is insufficient for reliable interactive benchmarking. It proposes Calibrated User Embeddings (CUE), combining a hierarchical session encoder, a slot-conditioned decoder that generates behavioral and stylistic commands, retrieved training examples, and a latent diffusion sampler. Contrast-derived command supervision supports both retrospective replay of observed sessions and simulation from sampled representations, without directly optimizing outcome labels. Experiments on τ²-Bench, SimulatorArena, and PRISM evaluate fidelity, success alignment, and failure reproduction across simulator backbones. The distinctive contribution is the integration of reusable latent conditioning with a detailed outcome-oriented evaluation, although the benefits attributable specifically to representation learning and diffusion remain unresolved.

Critical strengths
  1. The evaluation addresses an important distinction between realistic dialogue and valid benchmarking. Comparing fidelity metrics with aggregate success, paired outcomes, and failure composition reveals that improvements in naturalness do not reliably imply better agreement with human outcomes. Reporting the competitive no-persona baseline makes this observation particularly useful.
  2. The framework offers a practical interface across simulator models. Decoding representations into natural-language commands permits reuse with black-box simulators without retraining CUE, including an evaluated model family absent from the annotation contrast set. There are meaningful system-level improvements: with Gemini, replay outcome F1 increases from 0.56 for None to 0.62, while document-quality MAE improves over RealUserSim across all three backbones in Table 4.
  3. The paper provides substantial analysis and appropriately qualifies its claims. The appendices include uncertainty intervals, conditional evaluation counts, a detailed failure taxonomy, and pairing and sampling comparisons. The authors explicitly distinguish sessions from individuals, retrospective reconstruction from prospective prediction, and full-system performance from evidence about individual components.
Critical weaknesses
  1. The experiments do not isolate the contribution of the learned representation. CUE receives eight retrieved training examples, whereas replay RealUserSim receives none; supervision and sampling corpora also differ, and Appendix C.4 explicitly acknowledges the missing matched-example control. Comparisons against direct persona extraction using the same data and examples, together with command-only and retrieval-only CUE variants, are necessary to determine whether the gains arise from the proposed representation or from the surrounding annotation and retrieval pipeline.
  2. The evidence for user-specific outcome reproduction remains confounded by completed-session information. Replay encodes the entire target trajectory, including assistant context, while the shuffled-CUE experiment changes representations across tasks and evaluates mimicry rather than outcome agreement. Although the retrospective scope is clearly disclosed, prefix-only conditioning and within-task persona swaps would better establish whether CUE captures transferable behavioral variation rather than information about the realized task interaction.
Questions for authors
  1. Do CUE’s gains persist under matched supervision and example access? Please compare against direct persona extraction with the same source data and retrieved examples, and provide command-only/retrieval-only controls. Persistent gains would materially strengthen the methodological contribution; comparable performance from simpler controls would suggest that annotation and retrieval account for most of the improvement.
2 comments

We thank the reviewer for recognizing the importance of outcome calibration and the practical value of CUE’s model-agnostic conditioning. We agree that the original submission would benefit from matched-example controls, direct persona-generation baselines, and experiments separating user-specific behavior from outcome leakage from completed trajectories. To address these concerns, we conducted additional evaluations on Tau2-Bench on GPT 5.4 Mini and Gemini 3.5 Flash Lite (Llama was omitted due to time constraint).

Comparisons with matched retrieval and direct persona extraction.

We first evaluate CUE as from the paper (CUE commands + CUE examples), CUE commands only, CUE retrieved examples only, alongside RealUserSim augmented with the same CUE-retrieved examples (RealUserSim commands + CUE examples). We also evaluate direct behavioral manuals generated using CUE’s contrastive annotation procedure on the Tau-USI dataset and personas generated from Tau-USI target trajectories by Fable 5.2.

MethodGPT S2R-NGemini S2R-NGPT Outcome F1Gemini Outcome F1
CUE Commands + CUE Examples0.510.650.600.52
CUE Commands0.520.580.560.54
CUE Examples0.310.570.580.56
RealUserSim Commands0.330.360.550.57
RealUserSim Commands + CUE Examples0.370.490.540.55
Direct contrastive manuals0.490.630.600.54
Fable 5.2 trajectory personas0.530.820.530.58

Full CUE retains higher naturalness than RealUserSim with the same retrieved examples on both bases. Paired S2R-N differences (RealUserSim minus CUE) are −0.142 [−0.172, −0.113] on GPT and −0.163 [−0.194, −0.133] on Gemini. Thus, retrieved examples alone do not explain CUE's naturalness advantage.

The direct contrastive-manual baseline performs comparably to CUE on naturalness and outcome F1, while Fable 5.2 produces substantially stronger mimicry. Fable's AVA improvements over CUE are +0.194

+0.138,+0.251+0.138, +0.251

on GPT and +0.275

+0.230,+0.321+0.230, +0.321

on Gemini. Importantly, Fable's higher mimicry does not consistently improve calibration: its outcome F1 is lower than CUE's on GPT (0.53 versus 0.60; paired difference −0.063 [−0.115, −0.010]), but higher on Gemini (0.58 versus 0.52). This is again evidence in support of our primary thesis that fidelity and outcome calibration must be considered adjacently to one another, rather than assuming that one begets the other.

Strong direct persona extraction is a viable alternative, and we do not claim that learned representations universally outperform it, but CUE instead provides an amortized mechanism for encoding observed interactions into reusable behavioral representations and generating commands without repeating the expensive contrastive annotation procedure for each new interaction and a method of sampling new personas that do not necessarily appear in an existing dataset.

Contribution of the learned representation.

To investigate the representation itself, we trained CUE variants without auxiliary representation-learning losses and without the command-slot architecture, generating the whole prompt at once.

MethodGPT S2R-NGemini S2R-NGPT Outcome F1Gemini Outcome F1
CUE0.510.650.600.52
CUE w/o representation losses0.440.560.560.54
CUE w/o command slots0.460.570.540.54

Removing representation losses decreases S2R-N by 0.076 [0.053, 0.098] on GPT and 0.093 [0.067, 0.119] on Gemini. Removing command slots decreases S2R-N by 0.056 [0.033, 0.080] and 0.079 [0.052, 0.107], respectively. All four paired 95% intervals for naturalness differences exclude zero. These ablations support the contribution of the representation-learning objectives and slot-based decoding to naturalness.

Outcome leakage from encoding full trajectory conditioning,

We agree that encoding the completed trajectory, including assistant context, can introduce information correlated with the realized outcome. To test dependence on this information, we evaluate restricting user conditioning to the first 25% of user turns or only the first user turn.

MethodGPT AVAGemini AVAGPT Agent TVD ↓Gemini Agent TVD ↓GPT Outcome F1Gemini Outcome F1
All Turns0.610.640.330.270.600.52
First 25% of Turns0.540.640.290.300.570.55
First Turn0.290.370.310.360.610.58

First-turn conditioning substantially reduces mimicry: paired AVA differences relative to full CUE are −0.320 [−0.378, −0.259] on GPT and −0.270 [−0.339, −0.200] on Gemini. In contrast, outcome F1 changes by +0.017

−0.037,+0.070−0.037, +0.070

and +0.055

+0.002,+0.110+0.002, +0.110

, respectively. Agent TVD differences also have paired intervals including zero.

These results indicate that access to the completed trajectory is not necessary to obtain comparable outcome alignment in this evaluation, even though it contributes substantially to user-specific mimicry.

In the final version, we will update the Appendix to include all of these comparisons with multiple seed runs for all simulator bases.

Reviewer_VbaUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Existing LLM user simulator benchmarks typically evaluated on surface fidelity such as naturalness, stylistic mimicry and population diversity. This paper noticed this limitation and argued the importance of outcome calibration where the simulated interactions should reproduce real-user success rates, failure attributions and specific agent failure modes. The authors also proposed to train a novel model-agnostic user simulation called Calibrated User Embeddings without task rewards or failure labels. CUE archives a good fidelity as well as good outcome alignment compared to persona-based baselines.

Critical strengths

The paper is well-motivated with good observations and novelty. Distinguishing user fidelity from outcome calibration addresses a missing key point in interactive benchmarking. Their annotated human-in-the-loop 16-mode failure taxonomy also cleanly separates simulator derailment/premature stops from true agent errors, serving as a good resource. The diagnostic audit of existing fidelity metrics is also inspiring.

CUE can decode continuous representations into natural-language behavioral rules, which transfers zero-shot across unseen simulator families and held-out domains. So the method has good robustness and also scales positively with stronger simulator LLMs instead of suffering from the previously-noticed assistant bias (e.g., "inverse scaling").

Critical weaknesses

The experiment misses component ablations. For example, CUE augments decoded commands with 8 retrieved training utterances, whereas RealUserSim in replay does not have any example utterances and they didn't run retrieval-matched RealUserSim control. It is unclear how much of CUE's advantage comes from few-shot retrieved style examples rather than the learned continuous representation.

Also there might be bias in training/testing distribution match. CUE is trained on mixture of task-oriented customer-service datasets which may close to the τ2\tau ^2-Bench (e.g., airline and retail domains), however, baselines are trained more from the chat domain (e.g., RealUserSim uses WildChat and USP trains on LMSYS-Chat).

CUE also has a complex model stack including a dual-contrast annotation, 8 gates, hierarchical encoder, 10 loss weights in Table 6, slot-matched cross-attention decoder, and a 162M FiLM diffusion sampler. There are no ablations about removing individual representation losses, changing number of gates, comparing the encoder–decoder against directly prompting with LLM-extracted contrastive manuals, or if diffusion is necessary over bank resampling, etc.

Questions for authors

CUE encodes the whole target episode in user-conditioned replay. Is it possible that the task outcome can also leak in the replay?

3 comments

We thank the reviewer for recognizing the value in our thesis regarding outcome calibration, and our contributions of both the failure taxonomy and a generalizable simulator method. We agree that the original submission would benefit from stronger component ablations and more carefully controlled baseline comparisons. To address these concerns, we conducted additional evaluations on Tau2-Bench on GPT 5.4 Mini and Gemini 3.5 Flash Lite (Llama was omitted due to time constraint). Ablations include both inference ablations such as prefix-only user conditioning and command/example only prompting as well as training ablations such as removing representation losses and the slot-based decoding.

Baseline fairness

We agree that CUE’s eight retrieved examples could confound comparisons against RealUserSim. We therefore conducted three additional controls, introducing a retrieval-matched ReaUserSim baseline that receives the same CUE-retrieved examples while retaining the RealUserSim persona-generation procedure (RealUserSim commands + CUE examples).

MethodGPT S2R-NGemini S2R-NGPT Outcome F1Gemini Outcome F1
CUE commands + CUE examples0.510.650.600.52
CUE commands0.520.580.560.54
CUE examples0.310.570.580.56
RealUserSim commands0.330.360.550.57
RealUserSim commands + CUE examples0.370.490.540.55

Full CUE achieves higher naturalness than retrieval-matched RealUserSim on both bases. The paired S2R-N differences (RealUserSim minus CUE) are −0.142 [−0.172, −0.113] for GPT and −0.163 [−0.194, −0.133] for Gemini. These results demonstrate that CUE's naturalness advantage is not explained solely by access to retrieved examples.

Training distribution and domain matching.

We acknowledge that differences between CUE's task-oriented training data and the general-chat corpora used by some baselines introduce a potential distributional confound. Our additional τ-USI-conditioned RealUserSim evaluations partially address this concern by extracting personas from task-domain interactions, while the retrieval-matched variant additionally receives CUE's retrieved demonstrations. These controls do not equalize the underlying training corpora, and we will explicitly acknowledge this limitation. Our existing out-of-domain evaluations on SimulatorArena and PRISM provide complementary evidence that CUE's performance is not restricted to the task-oriented evaluation domains.

Architectural ablations.

We agree that it would be valuable to provide architectural training ablation to demonstrate the need for each component. We trained two additional CUE variants for this rebuttal period: no representation losses (which removes all auxiliary representation-learning objectives, keeping only the command supervision) and no command slots, which removes the slot-based architecture and generates instead a complete behavioral manual. We additionally evaluate the performance of prompts generated in the same manner as the training data via human and simulator contrasts by an LLM, using Tau-USI trajectories.

MethodGPT S2R-NGemini S2R-NGPT Outcome F1Gemini Outcome F1
CUE0.510.650.600.52
CUE w/o representation losses0.440.560.560.54
CUE w/o command slots0.460.570.540.54
Direct contrastive manuals0.490.630.600.54

We see that both architectural simplifications reduce naturalness, with the following paired differences relative to CUE:

AblationGPT Δ S2R-N [95% CI]Gemini Δ S2R-N [95% CI]
No representation loss-0.076
−0.098,−0.053-0.098, -0.053

| -0.093

−0.119,−0.067-0.119, -0.067

| | No command slots | -0.056

−0.080,−0.033-0.080, -0.033

| -0.079

−0.107,−0.052-0.107, -0.052

|

All four intervals exclude zero, providing evidence that both auxiliary representation learning and the slot-based architecture contribute to naturalness.

We also compare against using prompts generated via the original annotation pipeline. This control directly addresses the reviewer's suggestion to compare CUE against LLM-extracted contrastive manuals. The results are competitive with CUE, indicating that direct manual generation is a viable alternative when the required contrastive examples and annotation resources are available. CUE instead amortizes this process into a reusable encoder–decoder that produces behavioral commands without repeating the expensive contrastive annotation procedure at inference time and allowing for sampling of personas without a real trajectory. We will address this in the final paper and narrow claims about individual architectural components accordingly

Diffusion vs. bank resampling

We compare uniform training bank resampling (sampling persona prompts from the training set as generated using the contrastive method) and neighborhood diffusion (sampling using the diffusion sampler while providing 8 dataset samples to direct the sampler to the same user space as the original dataset) in addition to the unconditional diffusion sampling from the original paper.

MethodGPT S2R-NGemini S2R-NGPT S2R-CGemini S2R-C
Unconditional diffusion0.560.740.850.86
Uniform training bank resampling0.470.590.830.82
Neighborhood diffusion0.490.630.840.84

Unconditional diffusion consistently improves naturalness over uniform bank resampling (GPT: +0.096 [0.067, 0.124]; Gemini: +0.145 [0.111, 0.179]). It also produces modest improvements in S2R-C, although SD-C remains largely comparable. Outcome-calibration differences are mixed, suggesting that diffusion's primary demonstrated benefit is in sampled fidelity and coverage rather than calibration.

Task outcome leakage.

We agree that encoding the complete target interaction has the capacity to expose information correlated with the eventual outcome. To evaluate whether this drives CUE’s performance, we perform ablations using prefix-only conditioning, using either the first 25% of turns or only the first user turn to generate the CUE.

MethodGPT AVAGemini AVAGPT Agent TVD ↓Gemini Agent TVD ↓GPT Outcome F1Gemini Outcome F1
All Turns0.610.640.330.270.600.52
First 25% of Turns0.540.640.290.300.570.55
First Turn0.290.370.310.360.610.58

We find that access to the complete target trajectory is not required to achieve comparable outcome alignment. For example, restricting conditioning to the first user turn changes outcome F1 by +0.017

−0.037,+0.070−0.037, +0.070

on GPT and +0.055

+0.002,+0.110+0.002, +0.110

on Gemini. Agent TVD also remains within the paired task-bootstrap intervals, including for the largest observed difference (Gemini: +0.095

−0.075,+0.227−0.075, +0.227

). These findings indicate that CUE's outcome alignment does not depend entirely on information available only at the end of the interaction. We will clarify that full-trajectory conditioning is intended for retrospective user replay, whereas prefix conditioning more closely approximates prospective simulation.

In the final version, we will update the Appendix to include all of these comparisons with multiple seed runs for all simulator bases.

Thanks a lot for the detailed response and the additional experiments. I carefully reviewed them. My current score is good enough so I plan to maintain my score. I strongly encourage the authors to add those discussions into your final version.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Full Rebuttal Run Metrics

We thank the reviewers for their helpful feedback, especially on useful inference and training ablations. We attempted to run as many as possible during the rebuttal period to demonstrate our findings and to reinforce that we will include the full suite of ablations in the final camera-ready version.

Here are the full metric tables for the runs done during the rebuttal period. Relevant parts are included in the individual responses based on reviewer feedback. Once Llama runs are completed and multiple seeds are done, these will be incorporated into the Appendix. *Methods that were already present in the original paper.

AuthorsOpenReview ↗

Fidelity Metrics

User-Conditioned Fidelity

MethodS2R-NTT ↓AVAPT-3
GPT 5.4 Mini
CUE (Commands + Examples)*0.510.000.610.40
CUE (Commands Only)0.520.090.600.40
CUE (Examples Only)0.310.200.410.39
CUE (Prefix - 25% of Turns)0.520.140.540.38
CUE (Prefix - First Turn)0.400.350.290.35
CUE (Trained w/o Representation Losses)0.440.070.620.41
CUE (Trained w/o Slots)0.460.110.640.40
Contrastive Training Prompt0.490.120.680.42
Fable 5.2 Generated Prompt0.530.200.800.55
RealUserSim (Commands + CUE Examples)0.370.150.550.43
RealUserSim (Commands Only)*0.330.210.490.41
Gemini 3.1 Flash Lite
CUE (Commands + Examples)*0.650.070.640.41
CUE (Commands Only)0.580.040.670.41
CUE (Examples Only)0.570.050.600.37
CUE (Prefix - 25% of Turns)0.660.110.640.40
CUE (Prefix - First Turn)0.460.290.370.38
CUE (Trained w/o Representation Losses)0.560.050.710.40
CUE (Trained w/o Slots)0.570.010.680.38
Contrastive Training Prompt0.630.050.700.44
Fable 5.2 Generated Prompt0.820.150.920.64
RealUserSim (Commands + CUE Examples)0.470.090.820.52
RealUserSim (Commands Only)*0.360.050.650.37

Sampled Fidelity

MethodS2R-NTT ↓S2R-CSD-C
GPT 5.4 Mini
CUE (Unconditioned Diffusion)*0.560.070.850.59
CUE (Uniform Training-Bank Sampling)0.470.180.830.58
CUE (Neighborhood Diffusion)0.490.160.840.60
RealUserSim (Wildchat Commands + Examples)*0.390.250.590.58
RealUserSim (Wildchat Commands Only)0.370.240.460.57
Gemini 3.1 Flash Lite
CUE (Unconditioned Diffusion)*0.740.040.860.58
CUE (Uniform Training-Bank Sampling)0.590.110.820.58
CUE (Neighborhood Diffusion)0.630.090.840.58
RealUserSim (Wildchat Commands + Examples)*0.500.220.560.58
RealUserSim (Wildchat Commands Only)0.470.260.280.56
AuthorsOpenReview ↗

Outcome Calibration Metrics

User Conditioned Calibration

MethodUser Gap |Δ| ↓Attribution F1Agent TVD ↓Failure Mode F1Success Rate |Δ| ↓Outcome F1
GPT 5.4 Mini
CUE (Commands + Examples)*0.090.370.330.270.090.60
CUE (Commands Only)0.220.320.360.310.150.56
CUE (Examples Only)0.070.290.210.180.100.58
CUE (Prefix - 25% of Turns)0.130.310.290.240.070.57
CUE (Prefix - First Turn)0.060.390.310.170.100.61
CUE (Trained w/o Representation Losses)0.110.350.280.250.100.56
CUE (Trained w/o Slots)0.130.350.390.140.140.54
Contrastive Training Prompt0.050.500.300.230.090.60
Fable 5.2 Generated Prompt0.050.360.300.300.150.53
RealUserSim (Commands + CUE Examples)0.110.430.340.210.110.54
RealUserSim (Commands Only)*0.050.330.280.190.100.55
Gemini 3.5 Flash Lite
CUE (Commands + Examples)*0.050.280.270.360.130.52
CUE (Commands Only)0.060.330.340.220.130.54
CUE (Examples Only)0.070.320.380.360.140.56
CUE (Prefix - 25% of Turns)0.090.270.300.360.140.55
CUE (Prefix - First Turn)0.070.410.360.230.140.58
CUE (Trained w/o Representation Losses)0.070.310.360.150.140.54
CUE (Trained w/o Slots)0.040.290.280.320.190.54
Contrastive Training Prompt0.070.280.360.190.150.54
Fable 5.2 Generated Prompt0.050.360.350.190.130.58
RealUserSim (Commands + CUE Examples)0.080.250.320.270.160.55
RealUserSim (Commands Only)*0.100.270.440.190.200.57

Sampled Calibration

MethodUser Gap |Δ|↓Agent TVD ↓Success Rate |Δ| ↓
GPT 5.4 Mini
CUE (Unconditioned Diffusion)*0.090.280.09
CUE (Uniform Training-Bank Sampling)0.090.250.08
CUE (Neighborhood Diffusion)0.140.330.12
RealUserSim (Wildchat Commands + Examples)*0.160.230.12
RealUserSim(Wildchat Commands Only)0.080.230.08
Gemini 3.5 Flash Lite
CUE (Unconditioned Diffusion)*0.130.290.14
CUE (Uniform Training-Bank Sampling)0.050.400.14
CUE (Neighborhood Diffusion)0.070.380.15
RealUserSim (Wildchat Commands + Examples)*0.120.320.16
RealUserSim(Wildchat Commands Only)0.070.380.14

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.