acceptodds
Under review as a conference paper at ICLR 2027

Toward Valid Agent Evaluation: Reconciling Task Fidelity and Human Plausibility in LLM-Based Interactive User Simulation

Abstract

LLM-based user simulators enable scalable evaluation of interactive agents, but task success is informative only when the simulated interaction itself provides valid evidence of agent capability. This can fail in two ways: the simulator may drift from the assigned task or it may exhibit behavior that is extreme relative to real users. These risks motivate our joint assessment of Task Fidelity (TF) and Human Plausibility (HP). We audit 30 user LLMs, finding that 34.1% of successful interactions contain HP violations and 45.9% of failures contain TF violations. An intuitive response is to prompt simulators with personas, which we find improves HP but often at the expense of TF. To address the risks, we propose a method that separates what the simulator needs to accomplish from how it pursues those goals. Across 30 user LLMs, this separation turns average TF loss into a gain (−6.4 → +0.6 pp), while further improving HP (+6.1 pp). These gains extend to agent evaluation, with reduced score sensitivity to user LLM choice, more consistent rankings, and strong agreement with external evaluations. Together, these findings show that task-faithful, human-plausible interactions provide a stronger basis for interpreting agent outcomes as evidence of capability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agent, Human-AI Interaction, User Simulator

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bpxjOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

LLM-simulated users make agent evaluation scalable, but an agent’s success or failure only means something if the simulated interaction itself is valid. The authors identify two ways this can break: the simulator may drift from its assigned task, or it may behave unlike real users. They formalize these as Task Fidelity, judged by LLM judges, and Human Plausibility, checked against thresholds calibrated on real human dialogues. Auditing 30 user LLMs on τ²-bench, they find that many agent successes involve implausible user behavior, many failures involve the simulator drifting off task, and swapping the user LLM alone noticeably shifts agent scores. Prompting the simulator with a persona makes it more human-like but often costs task fidelity. To resolve this trade-off, the authors separate what the user needs to accomplish from how they pursue it: the task is compiled into a fixed structure, a Goal Tracker follows progress turn by turn, and a Persona Realizer decides each response according to a persona extracted from real human dialogues. Task fidelity returns to the no-persona level while human plausibility improves further. In agent evaluation, score differences are driven more by the agents themselves, rankings are more stable across user LLMs, and results remain consistent with VitaBench and Arena.

Critical strengths
  1. A well-posed problem. The paper argues that task outcomes only count as evidence when the simulated interaction itself is valid, and turns this into a testable diagnostic along two axes, TF and HP. The framework is simple and actionable, and the audit shows clearly how much benchmark scores alone can hide.

  2. A careful and transparent HP design. Thresholds are calibrated only on external human corpora, never on simulator results, which rules out tuning toward any method. The authors also test false-positive rates and sensitivity to the reference, and are upfront that passing HP does not mean being human-like, and that external references do not stand in for real-user performance.

  3. Solid experiments. Ablations cover module roles, inputs, call structure, and how the task is compiled, with comparisons against several existing methods and a trained simulator, backed by multiple human evaluations. The conclusions are well supported and directly useful to benchmark designers.

Critical weaknesses
  1. The claim that TPS is more plausible than T+P is under-supported. Most baseline HP violations come from overly long openings and turns, and TPS tells the generator to act only on the current goal and write one line per turn, which targets exactly these violations. Ablations show the Goal Tracker alone, without any persona, gets most of the HP gain, and persona prompting alone also gets most of it. If several mechanisms can push HP up this easily, TPS's lead on it says little. Under independent human-likeness measures, TPS's advantage over T+P is not consistent.

  2. Stable is not the same as valid. Removing simulator-caused failures is fine. But agent failures drop noticeably under TPS, and part of that drop is not explained by fewer TF violations. If that part comes from users being more cooperative than real people, the sim2real gap prior work points out, then however stable the rankings are, they measure agents facing friendly users and may not transfer to real ones. Neither external reference measures real-user task success, so this possibility can't be ruled out yet.

  3. TF measurement carries a bias risk specific to this method. In the main experiments, only TPS uses the compiled task structure, and the model that compiles it is also one of the TF judges. Under the rule that a rollout passes if any judge passes it, any self-preference by that judge would favor TPS alone. All three candidate judges are also user LLMs in the evaluated pool. The paper reports no human–judge TF agreement on TPS rollouts, so its central TF claim lacks targeted human validation.

Questions for authors
  1. If opening length and words per turn are removed from HP, does TPS still outperform T+P? This would show whether its HP edge over T+P comes from more than just shorter turns.

  2. The key question for validity is whether TPS users are more cooperative than real ones. Using the same agent as in the in-domain human dialogues, how does agent success under each method compare with that of real users on the same tasks?

  3. GPT-5.6-Terra both compiles the TPS task structure and serves as one of the TF judges. Do the 500 human-labeled dialogues used to select the judge include TPS rollouts? If so, what is human–judge agreement on them; if not, do the TF comparisons across methods still hold when combining only DeepSeek and Kimi?

6 comments
AuthorsOpenReview ↗

Response to Reviewer bpxj (1/3)

Thank you for highlighting the strengths of our problem formulation, transparent HP design, and thorough experimental validation. Below, we respond to your questions and feedback:

1. (W1, Q1) Whether TPS's HP advantage comes from shorter turns

W1-1. "Most baseline HP violations come from overly long openings and turns, and TPS tells the generator to write one line per turn."

Q1. "If opening length and words per turn are removed from HP, does TPS still outperform T+P?"

Thank you for the suggested analysis. Below, we show that the gain on Human Plausibility does not come from the length instruction, through the following three points:

1. HP without the length rules. We recompute HP without the two length-related rules, words per turn and opening (length-excluded HP), alongside HP over all seven rules (full HP). TPS still scores higher than T+P on length-excluded HP (89.9 vs. 87.9; higher in 19/30 user LLMs; Table A). The two rules are key signals of non-human-like behavior, but the other rules also capture it, and TPS lowers violations on most of the other rules as well (Table B).

Table A. HP pass rate (%, 30 user LLMs)

HP rulesTOT+PTPSModels TPS > T+Pp
full HP59.484.289.221/30.006
length-excluded HP78.387.989.919/30.080

Table B. Per-rule violation rate (%, 30 user LLMs)

HP ruleTOT+PTPS
Opening length22.35.01.2
Words per turn18.04.60.0
Repetition10.86.08.9
Strategy pivots5.52.10.7
Identity confusion4.72.10.4
Front-loading2.11.70.1
Clarification questions0.20.10.1

2. A length instruction alone is not enough. To see whether a simple instruction ("write one line per turn") alone could have achieved this, we add this instruction to T+P for 7 user LLMs (Table C). Full HP rises (76.3 → 86.3) but stays below TPS (92.0), and this gap holds on length-excluded HP (89.8 vs. 93.4). Together with Tables A and B, this indicates that TPS's HP advantage is not an artifact of targeting the length rules, either in the metric (of HP verdict design) or in the prompt (of TPS).

Table C. T+P + one-line instruction (7 small/mid-scale user LLMs)

MethodTFfull HPlength-excluded HP
T+P64.076.384.8
T+P + one-line instruction63.8 (−0.2)86.3 (+10.0)89.8 (+5.0)
TPS78.2 (+14.2)92.0 (+15.7)93.4 (+8.6)

3. Short turns reflect how humans write. The design of keeping each turn short is itself not an arbitrary choice to shorten responses, but is grounded in how humans actually write in human–AI dialogue. Table D reports the mean number of sentences per user turn, segmented with NLTK's sent_tokenize, in the four human corpora used in the paper and in simulated dialogues. Humans write 1.27–1.41 sentences per turn, while simulators write far more: 4.14 under TO and 3.24 under T+P. TPS brings this to 1.40, effectively closing the gap to human users.

Table D. Sentences per user turn

Human: τ-benchHuman: PRISMHuman: ThoughtTraceHuman: ConvApparelSim: TOSim: T+PSim: T+P + one-lineSim: TPS
Sentences per user turn1.291.271.411.344.143.242.211.40

Opening length and front-loading are not simple length measures, but indicators of how users disclose information over a dialogue [1, 2]. The length-related rules were thus included to capture non-human-like ways of disclosing information, not surface features such as verbosity (Appendix D.2). We will include the length-excluded HP and sentence-level analyses in the revised version.


W1-2. "TPS's lead on HP says little. Under independent human-likeness measures, TPS's advantage over T+P is not consistent."

Our argument for TPS over T+P has never rested on HP alone. Rather, our key finding is that there exists a trade-off between TF and HP, as persona prompting raises HP but lowers TF, and an interaction is valid for agent evaluation only when both hold. What matters is thus reconciling this trade-off. Naive persona prompting (T+P) struggles to do so, whereas TPS achieves it, as reflected in the joint pass rate (T+P: 57.6, TPS: 68.3).

Regarding the independent human-likeness measures (Appendix Table 14), T+P and TPS differ little, but all of them show the same ordering as HP: TO < {T+P, TPS} < human. These measures thus support HP's verdict rather than contradict it. The small gap between T+P and TPS is expected, since the two methods share the same persona profiles and these measures mainly capture wording and style.


[1] Grice, Logic and Conversation, Syntax and Semantics 1975.

[2] Clark & Brennan, Grounding in Communication, Perspectives on Socially Shared Cognition 1991.

AuthorsOpenReview ↗

Response to Reviewer bpxj (2/3)

2. (W2, Q2) Whether TPS users are more cooperative than real users

W2. "... part of that drop is not explained by fewer TF violations. If that part comes from users being more cooperative than real people, ... they measure agents facing friendly users and may not transfer to real ones."

Q2. "Using the same agent as in the in-domain human dialogues, how does agent success under each method compare with that of real users on the same tasks?"

Thank you for this important point. We address it by examining where the gain in agent success arises, and by comparing directly with real user data.

1. Misleading agent failures. To test whether the part of the drop in agent failures not explained by fewer TF violations comes from more cooperative users, we examine where the gain in agent success under TPS arises. We divided the 30 user LLMs into two halves of 15 by their TF-violating failure rate under TO, i.e., the share of rollouts in which the agent fails while the user violates task fidelity.

Agent success (mean ± SD)TOTPSΔ (TPS − TO)
All user LLMs (30)66.4 ± 19.174.6 ± 11.2+8.2
Task-faithful user LLMs (upper half, 15)82.3 ± 7.680.5 ± 2.9−1.8
Task-drifting user LLMs (lower half, 15)50.6 ± 12.768.7 ± 13.3+18.1

Pooled over the 30 user LLMs, agent success is higher under TPS than under TO (66.4 → 74.6), but this does not mean that TPS makes the user simulator more cooperative. The two groups show where the gain arises:

  • Low TF-violating failure under TO: TPS < TO. Agent success decreases under TPS (82.3 → 80.5). TPS does not make the task easier for the agent.
  • High TF-violating failure under TO: TPS > TO. Agent success rises under TPS (50.6 → 68.7). The low score under TO reflects failures that cannot be attributed to the agent, not more cooperative users under TPS.

We further find that, across all TO rollouts, the agent fails 46.7% of the time when the user violates task fidelity, versus 23.6% when it does not. These failures, however, are attributed to the agent. Thus, the higher agent success under TPS reflects the removal of misleading agent failures, not more cooperative users.


2. Comparison with real users. To compare directly with real users, we follow the setting of [3]. This work reports the agent success rate with real users on τ-bench (retail and airline), using GPT-5.2, the agent used in the in-domain human dialogues. We run the same agent under this setting with DeepSeek-V4-Flash as the user LLM, average simulated results over three runs, and take real-user agent success from [3].

Agent successHP pass
Sim users (TO)75.0%51.2%
Sim users (T+P)75.6%86.0%
Sim users (TPS)70.1%93.3%
Real users63.6%97.2%

As simulated users become more human-like, agent success moves toward the real-user level. T+P raises the HP pass rate but leaves agent success unchanged. TPS is closest to real users on both measures, so it lowers agent success rather than inflating it.

However, a gap to real users still exists under TPS. Closing this sim2real gap remains an open goal, and we hope our work contributes toward it. We will add this real-user comparison and discuss the remaining gap as a limitation in the revised version.


[3] Zhou et al., Mind the Sim2Real Gap in User Simulation for Agentic Tasks, COLM 2026.

AuthorsOpenReview ↗

Response to Reviewer bpxj (3/3)

3. (W3) Potential bias in TF measurement

W3. "... only TPS uses the compiled task structure, and the model that compiles it is also one of the TF judges. ... any self-preference by that judge would favor TPS alone. All three candidate judges are also user LLMs in the evaluated pool."

We note the reviewer's concern, and address each potential source of bias below.

1. Self-preference by Terra. The TF judge bases its verdict only on the dialogue and the scenario. The dialogue is generated by the user LLM under evaluation, and the scenario is the original, not the compiled structure. Since no output from Terra reaches the judge, there is no channel through which self-preference could favor TPS. The following results confirm that TPS's TF gains do not come from self-preference by Terra:

  • Self-compilation Ablation (Table 21). We let each user LLM compile its own scenario instead of using Terra's structure. TPS keeps its TF: within 0.7 pp (large-scale) and 2.5 pp (small/mid-scale) of the Terra-compiled version, and still above T+P. TPS's TF thus does not depend on Terra as the compiler.
  • Judging without Terra. We rescore all rollouts without Terra (G), using DeepSeek (D), Kimi (K), or both, in place of the main judge G∨D, which agrees best with human labels (93.2% vs. 73.2–87.6%; Table 7, Appendix D.1). As shown in the table below, TPS retains the highest TF under every combination, and the method ordering is unchanged. TPS's TF thus does not depend on Terra as a judge.
TOT+PTPS
G∨D (main)73.467.074.0
D∨K70.263.571.1
D69.162.770.4
K66.860.967.5

2. Advantage of TPS from the compiled structure. In Table 22, we give TO and T+P the Terra-compiled structure in place of the original scenario. Their TF does not improve. TF gain of TPS therefore comes from its design, which separates task tracking from persona realization, not from the compiled input itself.

3. Self-preference by judges as user LLMs. Each judge model appears as a user LLM under all three methods (TO, T+P, TPS), so any self-preference would apply to all three alike. We verify this directly in the table below:

TOT+PTPS
All user LLMs (30)73.467.074.0
Without judge LLMs (27)71.965.072.7

Excluding the three judge models (Terra, DeepSeek-V4-Flash, Kimi-K2.6) leaves the TF pattern unchanged: T+P < TO ≲ TPS. The method comparison thus does not depend on including the judge models among the user LLMs. We will include these judge robustness analyses in the revised version.


4. (Q3) Human–judge agreement on TPS rollouts

Q3. "Do the 500 human-labeled dialogues used to select the judge include TPS rollouts? If so, what is human–judge agreement on them; ..."

In the human annotation for judge selection (Appendix D.1), the 500 dialogues are sampled evenly across TO, T+P, and TPS, with user LLMs drawn at random. We compare the judge with human labels on each subset:

TOT+PTPS
Agreement (%)92.893.493.4
Δ TF, judge − human (pp)+2.4+3.0+1.8

Agreement is consistent across methods, and the judge's TF exceeds human TF by no more on TPS than on TO or T+P. The judge thus does not inflate TF on TPS. We will add the composition of this set to Appendix D.1.


We hope our responses address your concerns, and we thank you again for your thoughtful feedback and questions, which have helped us improve the clarity and completeness of our paper.

Reviewer_5eUiOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper is about the validity of LLM-based user simulators for evaluating interactive agents. It proposes two evaluation metrics: Task Fidelity (TF) and Human Plausibility (HP). It shows that prompting simulators with human personas can improve HP but often reduces TF. The paper proposes Task–Persona Separation (TPS) to address this issue. TPSuses a Goal Tracker to maintain the assigned task and a Persona Realizer to generate human-like responses. The experiments show that TPS improves both TF and HP across different user LLMs.

Critical strengths
  1. It’s smart to separate task fidelity from human plausibility when it comes to LLM-based user simulations.
  2. The evaluation and ablations are extensive. The paper evaluates 30 user LLMs, multiple agent models, and different baselines.
  3. In addition to evaluating the quality of user simulators, the paper studies how their method affects agent rankings. This makes the results more meaningful.
Critical weaknesses
  1. It is unclear how much the improvements in HP reflect improvements in actual human-like behavior. Human Plausibility (HP) is mainly based on a few statistical indicators of user behavior, e.g., response length and repetition. Therefore, passing HP does not necessarily mean the simulator behaves like a real human. In fact, Table 14 shows that there is still a significant gap between simulated and human dialogues under other human-likeness metrics.
  2. Although TPS improves the consistency of agent rankings across different LLMs, it is unclear whether this means more accurate evaluation of agent capability with real users. Note that the external evaluation uses VitaBench and this benchmark measures general human preferences rather than task completion.
Questions for authors
  1. In Table 14, T+P performs better than TPS on some human-likeness metrics, despite TPS having a higher HP pass rate. How should these differences be interpreted?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 5eUi (1/2)

Thank you for highlighting the strengths of our decomposition of simulation validity, comprehensive empirical validation, and practical implications for agent evaluation. Below, we respond to your questions and feedback:

1. (W1) On whether HP improvements reflect human-like behavior

W1-1. "Human Plausibility (HP) is mainly based on a few statistical indicators of user behavior, e.g., response length and repetition."

We would like to clarify that Human Plausibility (HP) in our paper is not a direct metric of human similarity. As described in Sec. 2.2, we did not define HP to ensure the human-likeness of a simulator, but rather to detect its non-human-likeness, which undermines the use of an agent's score as evidence of its capability with real users.

A claim that a behavior "lies outside the human boundary" can easily be overstated if made without caution. So we approached it conservatively, which is why HP is restricted to a limited number of indicators:

  • Surface-level lexical and stylistic indicators are intentionally excluded (Sec. 2.2; Appendix D.2). This is because even if a user simulator is less human-like only superficially or lexically, there is no guarantee that the agent's outcome is invalidated as evidence of its capability.
  • Indicators are selected as conservatively as possible (Appendix D.2–D.3): under large-scale(13K) corpora of human dialogue and their statistical support, we used only indicators that can detect behaviors that humans would rarely exhibit (designed to flag at most 5% of human dialogues; observed to flag 2.83% of human dialogues).

For example, LLM simulators are generally more polite than real users, but politeness alone does not make a simulator human-implausible, as there exist a non-negligible number of users in real-world who stay polite throughout a conversation. In contrast, a user who writes essay-length messages of more than 58 words in nearly every turn is rarely observed among real users. Considering these principles, we could not directly adopt many of the indicators such as politeness and emotional expression, and as a result, only a small number of indicators remain.

In addition, to show that our HP verdict does not rely solely on a few indicators (e.g., response length or repetition), we examined the HP violation rates of TO, T+P, and TPS on each indicator separately.

IndicatorTOT+PTPS
Opening length22.35.0 (−78%)1.2 (−95%)
Words per turn18.04.6 (−75%)0.0 (−99.9%)
Repetition10.86.0 (−45%)8.9 (−18%)
Strategy pivots5.52.1 (−62%)0.7 (−86%)
Identity confusion4.72.1 (−55%)0.4 (−91%)
Front-loading2.11.7 (−21%)0.1 (−94%)
Clarification questions0.20.1 (−52%)0.1 (−64%)
Any (HP violation rate)40.615.8 (−61%)10.8 (−73%)

These results suggest that changes in HP are not driven by a single indicator but reflect changes across multiple behaviors, showing that our HP verdict captures diverse aspects of user behavior despite its small number of indicators. We will include this per-indicator analysis in the Appendix of the revised version.


W1-2. "In fact, Table 14 shows that there is still a significant gap between simulated and human dialogues under other human-likeness metrics."

Although our HP verdict is not a measure of human similarity, we compared it with existing human-likeness metrics to show that we did not devise an entirely unrelated measure, but lies along the same research line.

As the reviewer points out, persona-conditioned methods such as T+P and TPS have not yet reached the level of real humans. Here, we make two points.

  1. Compared with the simulator without persona conditioning (TO), both methods come substantially closer to the human level, suggesting that persona conditioning is one viable approach among several toward human-like user simulation.
  2. Nevertheless, the remaining gap suggests that user simulation, in which machines model human behavior, remains a long-term research challenge. We believe our validity criteria provide a foundation for this direction, and will discuss this remaining gap as a limitation in the revised version.
AuthorsOpenReview ↗

Response to Reviewer 5eUi (2/2)

2. (Q1) Interpreting the differences between T+P and TPS in Table 14

Q1. "In Table 14, T+P performs better than TPS on some human-likeness metrics, despite TPS having a higher HP pass rate. How should these differences be interpreted?"

Thank you for this question. We do not view these differences as contradicting our HP results, for the following two reasons:

1. The metrics measure different aspects. We would like to clarify that the human-likeness metrics in Table 14 are largely based on the text of user utterances (e.g., word n-grams or embedding distributions), whereas HP is based on the user's behavior over a multi-turn interaction. Moreover, their purposes differ: these metrics assess how similar a simulator is to humans, whereas HP detects whether the user's behavior falls outside the human range.

2. We do not claim that TPS is more human-like than T+P. Given these differences, we did not expect these metrics to clearly separate T+P and TPS, and accordingly, our paper does not analyze the differences between them on these external metrics. Nevertheless, we included them because they are widely used in prior work, and agreement with them in direction serves as an external check that our Human Plausibility verdict is not an isolated, self-defined criterion. Indeed, the tendency "TO < T+P/TPS < Human" is clear, and these metrics also correlate with the HP pass rate across simulators (Spearman ρ = 0.53–0.75 for five of the six text-based metrics and ρ = 0.68–0.86 for the behavior-based metrics in Table 14). This suggests that our HP verdict points in the same direction as these metrics, and that persona conditioning brings simulators closer to humans. We will incorporate this correlation analysis into the revised version.


3. (W2) On whether ranking consistency reflects agent capability with real users

W2. "Although TPS improves the consistency of agent rankings across different LLMs, it is unclear whether this means more accurate evaluation of agent capability with real users. Note that the external evaluation uses VitaBench and this benchmark measures general human preferences rather than task completion."

Thank you for this point. Agent capability with real users is the ultimate target of any simulator-based evaluation, and ranking consistency across user LLMs is a necessary step toward it rather than the end goal. This is why our evaluation goes beyond consistency (L453–454), as we elaborate below.

(1) Convergent validity. A measure can be trusted to capture its intended construct when it agrees with independent measures obtained through different methods (Campbell & Fiske, 1959). Figure 7(c) measures external agreement following this principle, comparing against VitaBench (task performance, LLM-simulated users) and Arena (real users' pairwise judgments). To further diversify the methods, we add MultiChallenge (Sirdeshmukh et al., 2025), which evaluates how well models handle realistic multi-turn conversations with users. As shown below (Spearman ρ with 95% CIs), TPS maintains strong agreement with all three references, including the newly added one. Consistent agreement across references that differ in method makes it unlikely that our rankings reflect an artifact of any particular evaluation setup. We will add the MultiChallenge results to the revised version.

ReferencenTOT+PTPS
VitaBench100.63 [0.30, 0.92]0.83 [0.66, 0.93]0.92 [0.88, 0.96]
Arena100.71 [0.38, 0.92]0.83 [0.63, 0.90]0.92 [0.86, 0.95]
MultiChallenge (new)100.73 [0.54, 0.94]0.77 [0.63, 0.91]0.81 [0.74, 0.88]

(2) Interaction-level validity. TPS also improves both interaction-level validity criteria we define: task fidelity, which ensures outcomes are attributable to the agent, and human plausibility, which ensures the agent is tested under conditions resembling real users. The improved consistency is thus obtained under interactions that are both task-faithful and closer to real users, bringing the evaluation a step closer to measuring agent capability with real users.


We hope our responses address your concerns, and we thank you again for your thoughtful feedback and questions, which have helped us improve the clarity and completeness of our paper.

[1] Campbell & Fiske, Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix, Psychological Bulletin 1959.

[2] Deshpande et al., MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs, ACL Findings 2025.

Dear Reviewer, Thank you for your valuable time and for reviewing our paper. As the discussion deadline approaches, we would be very grateful if you could take a moment to look over our response. We have thoughtfully addressed your concerns, and would greatly appreciate any further feedback.

Reviewer_dTcBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors tackle the challenge of building effective LLM-based user simulators. The authors identify two key bottlenecks in today's simulators: (1) simulators can drift from the assigned task, i.e. task fidelity must be a focus, or (2) simulator behavior may be extreme compared to real-world users, i.e. human plausibility must be a focus. The authors validate the existence of these challenges through their evaluations across 30 LLM-based user simulators. They propose a solution that separates what the user needs to accomplish from how it pursues goals, showing improvements across task fidelity and human plausibility.

Critical strengths

-The paper is clearly written and easy to follow.

-Interesting experimental finding about persona prompting, which is among the most popular approaches to LLM-based user simulators: persona prompting increases human plausibility but often at cost to task fidelity.

-Overall the approach of separating what the user needs to accomplish from how it pursues goals is intuitive and the experimental results validate the effectiveness.

Critical weaknesses

-I suggest simplifying Figure 4, it’s hard to follow.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to Reviewer dTcB

Thank you for highlighting the strengths of our clarity of presentation, empirical insights, and validated method design.

Regarding the suggestion on Figure 4, we agree that it is dense and its readability can be improved. Due to the page limit, the figure combined two multi-step procedures: persona extraction from human dialogue data, and the operation of our modules during multi-turn interaction between user simulator and agent. With the additional page in the revised version, we will split it into two figures and simplify each.

If any further questions or thoughts arise, please feel free to let us know during the discussion period. Thank you again for your feedback.

Dear Reviewer, Thank you for your valuable time and for reviewing our paper. As the discussion deadline approaches, we would be very grateful if you could take a moment to look over our response. We have thoughtfully addressed your suggestion, and would greatly appreciate any further feedback.

Comments on the paper

1 comment

We thank all reviewers for their feedback. We have uploaded a revised manuscript that incorporates the analyses and clarifications from our responses. Additions are marked in blue, and the main text now uses the additional page.

  • Figures 4–5 (Sec. 4). The former Figure 4 is split into two simplified figures, one for persona extraction and one for Task–Persona Separation. (Reviewer dTcB)
  • Sec. 5.2, Figure 8(c), Appendix F.4, Table 40. MultiChallenge is added as a third external reference. (Reviewer 5eUi)
  • Sec. 5.2, Table 2, Appendix F.5. Comparison of agent success with real users. (Reviewer bpxj)
  • Appendix A. The remaining gap between simulated and real users is discussed as a limitation. (Reviewers 5eUi, bpxj)
  • Appendix D.1, Tables 9–10. Composition of the 500 human-labeled dialogues, human–judge agreement by method, and TF under alternative judges and without the judge models among the user LLMs. (Reviewer bpxj)
  • Appendix D.2, Table 13. Per-indicator HP violation rates under TO, T+P, and TPS, and HP recomputed without the two length indicators. (Reviewers 5eUi, bpxj)
  • Appendix E.4, Tables 31–32. T+P with the one-line instruction, and sentences per user turn in human and simulated dialogues. (Reviewer bpxj)
  • Appendix F.2, Table 38. Agent success under TO and TPS by user-LLM group. (Reviewer bpxj)

We are happy to answer any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.