- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
LLM-simulated users make agent evaluation scalable, but an agent’s success or failure only means something if the simulated interaction itself is valid. The authors identify two ways this can break: the simulator may drift from its assigned task, or it may behave unlike real users. They formalize these as Task Fidelity, judged by LLM judges, and Human Plausibility, checked against thresholds calibrated on real human dialogues. Auditing 30 user LLMs on τ²-bench, they find that many agent successes involve implausible user behavior, many failures involve the simulator drifting off task, and swapping the user LLM alone noticeably shifts agent scores. Prompting the simulator with a persona makes it more human-like but often costs task fidelity. To resolve this trade-off, the authors separate what the user needs to accomplish from how they pursue it: the task is compiled into a fixed structure, a Goal Tracker follows progress turn by turn, and a Persona Realizer decides each response according to a persona extracted from real human dialogues. Task fidelity returns to the no-persona level while human plausibility improves further. In agent evaluation, score differences are driven more by the agents themselves, rankings are more stable across user LLMs, and results remain consistent with VitaBench and Arena.
Critical strengths
-
A well-posed problem. The paper argues that task outcomes only count as evidence when the simulated interaction itself is valid, and turns this into a testable diagnostic along two axes, TF and HP. The framework is simple and actionable, and the audit shows clearly how much benchmark scores alone can hide.
-
A careful and transparent HP design. Thresholds are calibrated only on external human corpora, never on simulator results, which rules out tuning toward any method. The authors also test false-positive rates and sensitivity to the reference, and are upfront that passing HP does not mean being human-like, and that external references do not stand in for real-user performance.
-
Solid experiments. Ablations cover module roles, inputs, call structure, and how the task is compiled, with comparisons against several existing methods and a trained simulator, backed by multiple human evaluations. The conclusions are well supported and directly useful to benchmark designers.
Critical weaknesses
-
The claim that TPS is more plausible than T+P is under-supported. Most baseline HP violations come from overly long openings and turns, and TPS tells the generator to act only on the current goal and write one line per turn, which targets exactly these violations. Ablations show the Goal Tracker alone, without any persona, gets most of the HP gain, and persona prompting alone also gets most of it. If several mechanisms can push HP up this easily, TPS's lead on it says little. Under independent human-likeness measures, TPS's advantage over T+P is not consistent.
-
Stable is not the same as valid. Removing simulator-caused failures is fine. But agent failures drop noticeably under TPS, and part of that drop is not explained by fewer TF violations. If that part comes from users being more cooperative than real people, the sim2real gap prior work points out, then however stable the rankings are, they measure agents facing friendly users and may not transfer to real ones. Neither external reference measures real-user task success, so this possibility can't be ruled out yet.
-
TF measurement carries a bias risk specific to this method. In the main experiments, only TPS uses the compiled task structure, and the model that compiles it is also one of the TF judges. Under the rule that a rollout passes if any judge passes it, any self-preference by that judge would favor TPS alone. All three candidate judges are also user LLMs in the evaluated pool. The paper reports no human–judge TF agreement on TPS rollouts, so its central TF claim lacks targeted human validation.
Questions for authors
-
If opening length and words per turn are removed from HP, does TPS still outperform T+P? This would show whether its HP edge over T+P comes from more than just shorter turns.
-
The key question for validity is whether TPS users are more cooperative than real ones. Using the same agent as in the in-domain human dialogues, how does agent success under each method compare with that of real users on the same tasks?
-
GPT-5.6-Terra both compiles the TPS task structure and serves as one of the TF judges. Do the 500 human-labeled dialogues used to select the judge include TPS rollouts? If so, what is human–judge agreement on them; if not, do the TF comparisons across methods still hold when combining only DeepSeek and Kimi?
6 comments
Response to Reviewer bpxj (2/3)
2. (W2, Q2) Whether TPS users are more cooperative than real users
W2. "... part of that drop is not explained by fewer TF violations. If that part comes from users being more cooperative than real people, ... they measure agents facing friendly users and may not transfer to real ones."
Q2. "Using the same agent as in the in-domain human dialogues, how does agent success under each method compare with that of real users on the same tasks?"
Thank you for this important point. We address it by examining where the gain in agent success arises, and by comparing directly with real user data.
1. Misleading agent failures. To test whether the part of the drop in agent failures not explained by fewer TF violations comes from more cooperative users, we examine where the gain in agent success under TPS arises. We divided the 30 user LLMs into two halves of 15 by their TF-violating failure rate under TO, i.e., the share of rollouts in which the agent fails while the user violates task fidelity.
| Agent success (mean ± SD) | TO | TPS | Δ (TPS − TO) |
|---|---|---|---|
| All user LLMs (30) | 66.4 ± 19.1 | 74.6 ± 11.2 | +8.2 |
| Task-faithful user LLMs (upper half, 15) | 82.3 ± 7.6 | 80.5 ± 2.9 | −1.8 |
| Task-drifting user LLMs (lower half, 15) | 50.6 ± 12.7 | 68.7 ± 13.3 | +18.1 |
Pooled over the 30 user LLMs, agent success is higher under TPS than under TO (66.4 → 74.6), but this does not mean that TPS makes the user simulator more cooperative. The two groups show where the gain arises:
- Low TF-violating failure under TO: TPS < TO. Agent success decreases under TPS (82.3 → 80.5). TPS does not make the task easier for the agent.
- High TF-violating failure under TO: TPS > TO. Agent success rises under TPS (50.6 → 68.7). The low score under TO reflects failures that cannot be attributed to the agent, not more cooperative users under TPS.
We further find that, across all TO rollouts, the agent fails 46.7% of the time when the user violates task fidelity, versus 23.6% when it does not. These failures, however, are attributed to the agent. Thus, the higher agent success under TPS reflects the removal of misleading agent failures, not more cooperative users.
2. Comparison with real users. To compare directly with real users, we follow the setting of [3]. This work reports the agent success rate with real users on τ-bench (retail and airline), using GPT-5.2, the agent used in the in-domain human dialogues. We run the same agent under this setting with DeepSeek-V4-Flash as the user LLM, average simulated results over three runs, and take real-user agent success from [3].
| Agent success | HP pass | |
|---|---|---|
| Sim users (TO) | 75.0% | 51.2% |
| Sim users (T+P) | 75.6% | 86.0% |
| Sim users (TPS) | 70.1% | 93.3% |
| Real users | 63.6% | 97.2% |
As simulated users become more human-like, agent success moves toward the real-user level. T+P raises the HP pass rate but leaves agent success unchanged. TPS is closest to real users on both measures, so it lowers agent success rather than inflating it.
However, a gap to real users still exists under TPS. Closing this sim2real gap remains an open goal, and we hope our work contributes toward it. We will add this real-user comparison and discuss the remaining gap as a limitation in the revised version.
[3] Zhou et al., Mind the Sim2Real Gap in User Simulation for Agentic Tasks, COLM 2026.
Response to Reviewer bpxj (3/3)
3. (W3) Potential bias in TF measurement
W3. "... only TPS uses the compiled task structure, and the model that compiles it is also one of the TF judges. ... any self-preference by that judge would favor TPS alone. All three candidate judges are also user LLMs in the evaluated pool."
We note the reviewer's concern, and address each potential source of bias below.
1. Self-preference by Terra. The TF judge bases its verdict only on the dialogue and the scenario. The dialogue is generated by the user LLM under evaluation, and the scenario is the original, not the compiled structure. Since no output from Terra reaches the judge, there is no channel through which self-preference could favor TPS. The following results confirm that TPS's TF gains do not come from self-preference by Terra:
- Self-compilation Ablation (Table 21). We let each user LLM compile its own scenario instead of using Terra's structure. TPS keeps its TF: within 0.7 pp (large-scale) and 2.5 pp (small/mid-scale) of the Terra-compiled version, and still above T+P. TPS's TF thus does not depend on Terra as the compiler.
- Judging without Terra. We rescore all rollouts without Terra (G), using DeepSeek (D), Kimi (K), or both, in place of the main judge G∨D, which agrees best with human labels (93.2% vs. 73.2–87.6%; Table 7, Appendix D.1). As shown in the table below, TPS retains the highest TF under every combination, and the method ordering is unchanged. TPS's TF thus does not depend on Terra as a judge.
| TO | T+P | TPS | |
|---|---|---|---|
| G∨D (main) | 73.4 | 67.0 | 74.0 |
| D∨K | 70.2 | 63.5 | 71.1 |
| D | 69.1 | 62.7 | 70.4 |
| K | 66.8 | 60.9 | 67.5 |
2. Advantage of TPS from the compiled structure. In Table 22, we give TO and T+P the Terra-compiled structure in place of the original scenario. Their TF does not improve. TF gain of TPS therefore comes from its design, which separates task tracking from persona realization, not from the compiled input itself.
3. Self-preference by judges as user LLMs. Each judge model appears as a user LLM under all three methods (TO, T+P, TPS), so any self-preference would apply to all three alike. We verify this directly in the table below:
| TO | T+P | TPS | |
|---|---|---|---|
| All user LLMs (30) | 73.4 | 67.0 | 74.0 |
| Without judge LLMs (27) | 71.9 | 65.0 | 72.7 |
Excluding the three judge models (Terra, DeepSeek-V4-Flash, Kimi-K2.6) leaves the TF pattern unchanged: T+P < TO ≲ TPS. The method comparison thus does not depend on including the judge models among the user LLMs. We will include these judge robustness analyses in the revised version.
4. (Q3) Human–judge agreement on TPS rollouts
Q3. "Do the 500 human-labeled dialogues used to select the judge include TPS rollouts? If so, what is human–judge agreement on them; ..."
In the human annotation for judge selection (Appendix D.1), the 500 dialogues are sampled evenly across TO, T+P, and TPS, with user LLMs drawn at random. We compare the judge with human labels on each subset:
| TO | T+P | TPS | |
|---|---|---|---|
| Agreement (%) | 92.8 | 93.4 | 93.4 |
| Δ TF, judge − human (pp) | +2.4 | +3.0 | +1.8 |
Agreement is consistent across methods, and the judge's TF exceeds human TF by no more on TPS than on TO or T+P. The judge thus does not inflate TF on TPS. We will add the composition of this set to Appendix D.1.
We hope our responses address your concerns, and we thank you again for your thoughtful feedback and questions, which have helped us improve the clarity and completeness of our paper.
Response to Reviewer bpxj (1/3)
Thank you for highlighting the strengths of our problem formulation, transparent HP design, and thorough experimental validation. Below, we respond to your questions and feedback:
1. (W1, Q1) Whether TPS's HP advantage comes from shorter turns
Thank you for the suggested analysis. Below, we show that the gain on Human Plausibility does not come from the length instruction, through the following three points:
1. HP without the length rules. We recompute HP without the two length-related rules, words per turn and opening (length-excluded HP), alongside HP over all seven rules (full HP). TPS still scores higher than T+P on length-excluded HP (89.9 vs. 87.9; higher in 19/30 user LLMs; Table A). The two rules are key signals of non-human-like behavior, but the other rules also capture it, and TPS lowers violations on most of the other rules as well (Table B).
Table A. HP pass rate (%, 30 user LLMs)
Table B. Per-rule violation rate (%, 30 user LLMs)
2. A length instruction alone is not enough. To see whether a simple instruction ("write one line per turn") alone could have achieved this, we add this instruction to T+P for 7 user LLMs (Table C). Full HP rises (76.3 → 86.3) but stays below TPS (92.0), and this gap holds on length-excluded HP (89.8 vs. 93.4). Together with Tables A and B, this indicates that TPS's HP advantage is not an artifact of targeting the length rules, either in the metric (of HP verdict design) or in the prompt (of TPS).
Table C. T+P + one-line instruction (7 small/mid-scale user LLMs)
3. Short turns reflect how humans write. The design of keeping each turn short is itself not an arbitrary choice to shorten responses, but is grounded in how humans actually write in human–AI dialogue. Table D reports the mean number of sentences per user turn, segmented with NLTK's
sent_tokenize, in the four human corpora used in the paper and in simulated dialogues. Humans write 1.27–1.41 sentences per turn, while simulators write far more: 4.14 under TO and 3.24 under T+P. TPS brings this to 1.40, effectively closing the gap to human users.Table D. Sentences per user turn
Opening length and front-loading are not simple length measures, but indicators of how users disclose information over a dialogue [1, 2]. The length-related rules were thus included to capture non-human-like ways of disclosing information, not surface features such as verbosity (Appendix D.2). We will include the length-excluded HP and sentence-level analyses in the revised version.
Our argument for TPS over T+P has never rested on HP alone. Rather, our key finding is that there exists a trade-off between TF and HP, as persona prompting raises HP but lowers TF, and an interaction is valid for agent evaluation only when both hold. What matters is thus reconciling this trade-off. Naive persona prompting (T+P) struggles to do so, whereas TPS achieves it, as reflected in the joint pass rate (T+P: 57.6, TPS: 68.3).
Regarding the independent human-likeness measures (Appendix Table 14), T+P and TPS differ little, but all of them show the same ordering as HP: TO < {T+P, TPS} < human. These measures thus support HP's verdict rather than contradict it. The small gap between T+P and TPS is expected, since the two methods share the same persona profiles and these measures mainly capture wording and style.
[1] Grice, Logic and Conversation, Syntax and Semantics 1975.
[2] Clark & Brennan, Grounding in Communication, Perspectives on Socially Shared Cognition 1991.